Internet Outages Timeline

On March 17, ThousandEyes observed an incident impacting Workday users attempting to access the service. Throughout the approximately one-hour disruption, users received a Workday-generated “service unavailable” message, indicating a potential issue with the application backend. Further pointing to an application-related issue, ThousandEyes didn’t see any network issues connecting to Workday’s frontend web servers during the disruption.

During an outage, viewing signals in isolation, instead of considering all data points holistically, can lead to incorrect assumptions about the cause. While it might be tempting to jump to conclusions about a disruption’s source—especially when the incident coincides with a major event that seems an obvious culprit, such as Workday’s R1 feature release—doing so can prompt inappropriate mitigation strategies and hinder resolution. In this case, the Workday outage ended up having no connection to the feature release, according to the company’s statement.

On February 26, ThousandEyes detected a service disruption affecting global users of Slack that lasted for over nine hours. Network connectivity to the application’s frontend servers remained intact; however, users attempting to access the application received server errors, suggesting a backend application issue.

Verifying the performance of an entire service delivery chain for a service that’s experiencing issues is crucial for pinpointing the specific fault domain. This understanding guides the actions your team takes to help ensure the continuity or recovery of the impacted service. Possible responses may include switching to an alternative system, implementing mitigations on your side, or deciding to do nothing and waiting for the issue to resolve itself.

On December 11, Meta services, including Facebook and Instagram, experienced an outage. ThousandEyes detected internal server errors and timeouts, which may indicate issues with Meta's backend services. During the incident, network connectivity to Meta’s frontend web servers appeared to remain unaffected.

Identifying commonalities during an outage can be helpful in determining the cause. For instance, when an outage impacts multiple services from the same company across various regions, it can help narrow down the potential issue. In such cases, it’s more likely to be linked to a backend service problem.

Microsoft users in multiple regions encountered difficulties with some of the company’s services, including Outlook Online, due to a prolonged outage. Initially, the incident manifested as intermittent timeout and application errors, but the outage’s scope increased about five hours after its start, and again about 10.5 hours in. During the outage, ThousandEyes observed various conditions, including server errors, timeouts, and packet loss.

Intermittent issues often show as slow performance, making them difficult to identify. ITOps teams need a clear performance baseline to spot deviations that indicate outages. When disruptions happen, quickly identifying the source is crucial, starting by ruling out non-issues. By leveraging this information with other data, teams can better understand the outage's cause and communicate effectively with users.

On October 1, Salesforce instances hosted in AWS experienced an outage in which customers encountered timeouts, slow performance, and inaccessibility of the service, indicating application-related issues.

When an outage occurs, it is crucial to quickly identify the fault domain. This not only helps determine who is responsible for resolution but also guides the steps that can be taken to address the issue.

On September 12, some users experienced issues reaching Microsoft services like Microsoft 365. During the approximately 1.5-hour incident, ThousandEyes observed significant network packet loss, as well as connection timeouts, within Microsoft’s network. From what ThousandEyes saw, the problems only impacted a subset of users connecting to Microsoft’s network via AT&T.

When an issue occurs, it's important to quickly determine who is affected—whether it's all users or a certain group. It's also helpful to identify if the issue is impacting any specific services. This will not only help determine who is accountable for resolution, but also what mitigation steps can be taken, such as moving to alternate systems or altering network entry points.

On August 5, Microsoft experienced an incident that affected the availability of LinkedIn for some users around the globe in a disruption that lasted a little over an hour. The outage manifested as elevated packet loss in Microsoft’s network, as well as DNS resolution timeouts and HTTP errors. ThousandEyes observed some residual network latency issues after the reported resolution; however, they did not appear to prevent users from interacting with LinkedIn services.

During the outage, the connectivity and timeout issues were intermittent and impacted users unevenly. Intermittent issues can be particularly time consuming to identify and address, especially when they involve various kinds of symptoms. It's important to analyze these symptoms collectively rather than in isolation to avoid a misdiagnosis.

On July 19, a software update issue with Crowdstrike's security software caused widespread outages for various organizations, including airlines, banks, and hospitals. CrowdStrike stated that the problem originated from a single configuration file, which led to a logic error, resulting in system crashes and blue screens of death (BSOD) on affected Windows systems. Due to the extensive range of impacted services and the various outage conditions affecting endpoints, servers, and applications, network issues were considered as a potential cause. While Internet connectivity problems are often the common thread behind simultaneous outages across multiple apps and services, in this case, that was not a factor.

While digital services rely heavily on networks, web-based apps, and cloud services, the recent CrowdStrike incident serves as a reminder that other influences are also at play. It's crucial to efficiently pinpoint the source of a disruption, and a key part of this process is identifying what isn't causing the problem. Ensuring a smooth digital experience involves more than just the network—customers need to consider everything from the device to the app.

On May 16, Salesforce experienced a disruption that intermittently impacted some customers’ ability to reach the service for more than four hours, caused by intermittent failures from a third-party DNS service provider.

ITOps teams may encounter challenges when trying to diagnose intermittent issues. It is crucial to thoroughly analyze all the data within your complete digital supply chain, including third-party dependencies, and to establish ongoing monitoring for convenient comparison of conditions.

ThousandEyes observed a global disruption of google.com on May 1 that lasted nearly an hour. During the incident, many users attempting to perform a search on google.com appeared to receive HTTP 502 Bad Gateway errors, indicating an issue reaching a backend service. The problem seemed to lie with the connectivity or linkage between google.com and the search engine, rather than with the search engine itself, as other services utilizing the search engine's functions appeared unaffected.

When issues arise, quickly determine the domain from which the issue stems (backend, front-end service, local environment, network connectivity, CDN, etc.) and assess the extent of the disruption. Identify which functions or services are impacted. This will allow you to make informed decisions about what actions to take to minimize the impact on your users—preferably before they are impacted at all.

Around 8:50 AM (UTC), ThousandEyes observed degraded availability for customers using Azure services that depend on Azure Front Door. These customers would have likely experienced intermittent degraded performance, latency, and/or timeouts when attempting to access services hosted in the United Kingdom South region, which appeared to be centered on nodes located in London, England.

Once you identify where an outage is occurring, it may be possible to route around it or to an alternate location to avoid disruption. Determine which location and which workloads your services are leveraging so that you can easily identify an alternate route if needed.

At approximately 1:20 PM (UTC) on March 26, ThousandEyes observed a global disruption of team workspace application Atlassian Confluence. The application began responding to access requests with 502 Bad Gateway server errors, suggesting an application backend issue. Network paths to the application’s frontend web servers, which are hosted in AWS, were clear throughout the incident. By 2:35 PM (UTC), the service had completely recovered.

Error messages can give you some clues about what went wrong, but without context, these codes are like breadcrumbs, leading you in a direction but not giving the full picture. Isolating the true root cause requires looking at a variety of factors in context, including any third-party providers that your application or network infrastructure rely on.

On March 5, ThousandEyes observed outage conditions in parts of Comcast’s network, which impacted the reachability of many applications and services, including Webex, Salesforce, and AWS. The nearly two-hour outage seems to have affected traffic as it traveled Comcast’s network backbone in Texas, including traffic coming from states such as California and Colorado.

When services you rely on experience issues, a good first question to investigate is whether the problem lies with you, that application, or a third-party provider. Then you can implement appropriate backup plans and remediation efforts.

During a more than seven-hour disruption, Microsoft Teams users around the globe experienced service failures that impacted their ability to use the collaboration tool. ThousandEyes didn’t observe any packet loss connecting to the Microsoft Teams edge servers; however, ThousandEyes did observe application layer failures that are consistent with reported issues within Microsoft’s network that may have prevented the service’s edge servers from reaching application components on the backend.

During the disruption, Microsoft Teams appeared partially available, but some users experienced issues with core functions like messaging and calling. As a result, some may have initially assumed the problem was on their end, not Microsoft’s. Taking a look at the full end-to-end service delivery chain matters for accurate troubleshooting.

Twitter was rendered unreachable for approximately 45 minutes after a Russian Internet and satellite communications provider blackholed traffic by announcing one of Twitter’s prefixes. BGP misconfigurations are not uncommon. However, they can be used to block traffic in a targeted way, and it’s not always easy to tell when the situation is accidental versus intentional.

While your company might have RPKI implemented to fend off BGP threats, it's possible that your telco won't. Something to consider when selecting ISPs.