Following on from the network outage experienced on Thursday 17th June 2021, we have now received the full Reason For Outage (RFO) report.
We apologise again for the inconvenience this outage caused and thank you for your patience while the resolution was achieved.
Please see the abridged notes below and the full RFO can be downloaded here.
Incident Start: 17/06/2021 16:29
Incident Resolved: 17/06/2021 18:26
Mass service outage affecting multiple BT xDSL services and a small number of TTB xDSL services. The TTB services were only briefly affected.
Our Network Provider received an influx of monitoring alerts into the Support ticket queue. The incident was declared as a P0 and the Major Incident protocol was initiated. Our Network Provider’s Technical Director and Network teams were already aware of the issue and investigating the root cause.
The uplink path between our Network Provider’s core wholesale DSL router in Telehouse North and MPLS core failed. TTB sessions quickly rerouted to alternative nodes on the Provider’s network. BT’s sessions failed to redirect to another node, due to a configuration change on the BT Network.
Our Network Provider resolved issue by replacing the device uplink. Our Network Provider separately addressed the BT Network change the following day.
On 18/06/2021 our Network Provider worked with BT’s engineers to make configuration changes to the RADIUS setup. This restored the designed failover behaviour to the network.
In order to mitigate against this type of failure in the future our Network Provider will now need to perform regular testing to mitigate against changes to the way that BT’s network interact with our Provider’s Network.
Should you experience any service-affecting issues at any time, please email firstname.lastname@example.org or complete our contact form selecting Support or Faults (Service Desk) as your reason for contact. This will automatically create a ticket for the team to pick up. Alternatively, you can call us on 020 8300 9495 (select Option 1 for the Service Desk).