Ticket Details

Subject: 100G Edmonton-Surrey, Vancouver-Calgary down
Category: Outage
TicketID: 20211115-002
Start Time: 2021-11-15 17:26:39 EST (2021-11-15 22:26:39 UTC)
End Time: 2021-12-13 19:17:56 EST (2021-12-14 00:17:56 UTC)


TICKET HISTORY:

=== Updated by: Richard Klinger on 2021-12-14 15:50:12 EST (2021-12-14 20:50:12 UTC) ===

Closed.

=== Updated by: Richard Klinger on 2021-12-13 19:36:32 EST (2021-12-14 00:36:32 UTC) ===

Edmonton - Surrey has been returned to service, concluding the restoration of services through BC.

=== Updated by: Joshua Muysson on 2021-12-06 11:45:54 EST (2021-12-06 16:45:54 UTC) ===

Over the weekend, one of the circuits between Vancouver and Calgary has been restored.
The other circuit between Vancvouver and Calgary is still down. No ETR as of yet.
Edmonton to Surrey has had work started, with an ETR of middle of this week.

=== Updated by: Joshua Muysson on 2021-12-01 12:12:49 EST (2021-12-01 17:12:49 UTC) ===

Contacted service provider. Another storm hit over the weekend adding more water to the area. Still no ETA at this time. Clear weather predicted for the coming weekend.

=== Updated by: Richard Klinger on 2021-11-26 08:57:20 EST (2021-11-26 13:57:20 UTC) ===

Less optimistic tone on call with service providers today and more weather on the way in BC.  No known service restoration time.

=== Updated by: Richard Klinger on 2021-11-24 11:45:14 EST (2021-11-24 16:45:14 UTC) ===

We are still receiving responses indicating that it's unknown when repairs will be completed however work has slowly begun to restore connections through damaged routes as access has opened.

Time to full restoration is still looking like weeks however there is some optimism that Edmonton-Surrey will be restored measuring in days.

=== Updated by: Richard Klinger on 2021-11-19 08:39:35 EST (2021-11-19 13:39:35 UTC) ===

The province of British Columbia has declared a provincial state of emergency. The current focus is on restoring the transportation network.  Service providers are not expecting to have site access until next week.

We will provide additional updates as the situation progresses.

=== Updated by: Richard Klinger on 2021-11-16 19:42:08 EST (2021-11-17 00:42:08 UTC) ===

Damaged sites in BC remain inaccessible.  Service provider has indicated that will likely take days before field teams are allowed to enter the region to assess damage.

We are investigating the possibility of an additional 100G to mitigate ongoing risk of outage.

=== Created by: Jun Jian on 2021-11-15 18:39:18 EST (2021-11-15 23:39:18 UTC) ===

Started at the time noted above, we have multiple 100G backbone circuits outages. The both 100G Vancouver-Calgary on canarie wavelength and the provider circuit, the 100G Surrey-Edmonton circuit are down now. The network traffic to BCNet and Seattle now is being rerouted via diversity path to Seattle.  The network service performance to BCNet network and to Seattle is experiencing degradation.  The providers have been aware of the situation and are working on to restore the circuit. Updates will be sent out as soon as information becomes available.