ThousandEyes actively monitors the reachability and performance of thousands of services and networks across the global Internet, which we use to analyze outages and other incidents. The following analysis of Virgin Media's service disruption on April 4, 2023, is based on our extensive monitoring, as well as ThousandEyes’ global outage detection service, Internet Insights.
[Apr 4, 2023, 2:00 PM PT]
On April 4th, 2023, Virgin Media UK (AS 5089) experienced two outages that impacted the reachability of its network and services to the global Internet. The two outages shared similar characteristics, including the withdrawal of routes to its network, traffic loss, and intermittent periods of service recovery.
Overview of Incidents
- The first incident began at approximately 00:30 UTC and appeared to coincide with a series of BGP route withdrawals by Virgin Media UK, which significantly constrained its availability to the global Internet.
- Without a route to its network, Internet and transit providers around the world would have simply dropped traffic destined to services and endpoints within Virgin Media UK’s network. Users would have experienced the drops as prolonged connection attempts that eventually failed.
- While most traffic loss observed throughout the incident was due to the lack of available BGP routes, some service providers were not impacted by the withdrawals and continued to successfully route traffic towards the Virgin Media UK network; however, in these cases, all traffic was dropped at Virgin’s network edge, suggesting the network operator was experiencing systemic duress (likely impacting its control plane), thus preventing it from ingesting and routing traffic within its network.
- Periodic advertisements and withdrawals took place throughout the outage, indicating potential mitigation efforts, including the removal of its most significant peer LGI-UPC (AS 6830), from paths to its network.
- Over the nearly 7 hour incident, brief periods of service restoration were interspersed throughout—approximately 130 minutes in total—with the longest period lasting 1 hour and 20 minutes, between 02:55 and 04:15 UTC.
- The initial outage eventually resolved at approximately 7:20 UTC and network services remained stable from that time until mid-afternoon, at which point the earlier observed behavior returned.
- The second incident was less prolonged than the first one, starting at around 15:20 UTC and resolving at approximately 17:30 UTC.
- The conditions surrounding the outage were the same as the initial event, with similar patterns of route withdrawals, traffic loss and brief periods of service restoration.
- Given that the initial incident began in a period of time typical of maintenance activities (half past midnight local time), it may have resulted from a change to the network state by the service provider.
- Recurrence of a near identical incident later in the day could indicate that the triggering mechanism for the first incident was either not fully understood or was not completely resolved.
Stay tuned for further updates as we unpack details of the incident and share our analysis.
[Apr 4, 2023, 12:30 PM PT]
On April 4th, 2023, Virgin Media UK experienced two outages that impacted the reachability of its network and services to the global Internet. The two outages shared similar characteristics, including the withdrawal of routes to its network, traffic loss, and intermittent periods of service recovery. The first incident took place between ~00:30 UTC and ~7:00 UTC, while the second began at ~15:20 UTC and resolved at ~17:30 UTC.
You can explore the outage within the ThousandEyes platform here (no login required).
