This is the Internet Report, where we uncover what’s working and what’s breaking on the Internet—and why. This week’s episode is brought to you by the letter “O” for outages — in particular, there were a number of broadband providers, globally, that suffered localized outages this past week. After we run down our top headlines, including a satellite provider rolling out managed SD-WAN, we take a look at outages in Comcast and AT&T’s networks. Make sure you join us next week to hear from Atif Khan, CTO at Alkira, as we talk about multi-cloud networking.
Find us on:
Finally, don’t forget to leave a comment here or on Twitter, tagging @ThousandEyes and using the hashtag #TheInternetReport.
Catch up on past episodes of The Internet Report here.
Follow Along with the Transcript
Angelique Medina:
This is the Internet Report, where we uncover what's working and what's breaking on the Internet and why? So the headlines this week are interesting because they are all about broadband providers, with the exception of one. So broadband providers did not have a good week last week. There was a number of outages, including one on June 22nd, which took place in India on the Reliance Jio Network.
Archana Kesavan:
Right. They kickstarted the week on Monday. And Jio Fiber was actually one of India's most anticipated fiber broadband services. And the reason was because it was very accessible to the masses, and that launched last year. But this outage that they experienced was almost a 24-hour outage started the afternoon on June 22nd, and Reliance Jio confirmed that a few cities in Northern India, including Delhi were affected by the outage, but Twitter feeds and other channels, they reported that the outage was nationwide. Think what made this worse was that Jio's customer care, the live chat feature, or even the phone support, they seemed to buckle under the pressure as well. So leaving customers really in the dark with no clarity on what happened and when it would be fixed.
Angelique Medina:
So on top of that, there was a broadband outage in the UK. So this was Virgin Media and they experienced an impact to two of their services. So currently not just their broadband, but it was also their TV services as well. And this was a pretty long-lasting outage, almost a day, right?
Archana Kesavan:
Yeah, almost a day. It started I believe at 8:00 AM, London time. And then by the time it was resolved and Virgin Media put out a statement that it was resolved, it was almost 5:30 PM.
Angelique Medina:
So this was like peak usage hours during the day, starting when everyone's about to get to work. And a few of our colleagues who are based in the UK noticed when they were having issues, they were also testing from their own homes, looking at the network path to some services they were using. And they noticed some packet loss in Virgin Media's network.
Angelique Medina:
So it was kind of apparent to them that there was clearly a problem and it was specific to Virgin Media. Not a lot of information came out about this, there are some reports and some statements, but it's not really clear like what the underlying cause was at this point.
Archana Kesavan:
Right. Right. And to add on to that, here in the US, AT&T and Comcast had problems with their own. And stay tuned for that, Angelique's going to jump into that in the under the hood section.
Angelique Medina:
Yeah. And then finally, we've talked a lot about kind of satellite services have come up recently. We talked about Starlink a few weeks ago, that’s SpaceX's satellite service. There was another satellite provider, I believe their name is SES and they announced that they have a managed SD-WAN service for their customers that basically enables their customers to have a more resilient kind of network. Maybe using satellite connectivity is just one transport and then maybe using fixed fiber at the same time. So I believe that they're rolling out Versa Networks as the SD-WAN platform, and then enabling our customers to kind of implement policies based on, it could be the application, it could be performance, or it could just be availability and then balance traffic cross those two transport links. Another interesting thing they had in there about performance.
Archana Kesavan:
We were talking about how these satellite providers are having really high latency and not having enough bandwidth? And it's interesting to have some numbers in there, about 150 milliseconds is what they say, depending on the type of satellite you're connecting to. Obviously seems much more reasonable than the claims that were made by SpaceX. Which you talk about that Angelique, a couple of weeks ago in our podcast as well.
Angelique Medina:
I mean, it's interesting because the reputation of satellite conductivity is just that. That it's slow and that you don't have a lot of bandwidth and you're just not going to get a great performance. And so it seems like, there's a couple of different players in space that are now saying that they are able to offer an improved service. Now, in the case of SpaceX, are they able to achieve that? It would be interesting to see if they are able to. And they've actually launched a beta program for some users in the Northern hemisphere to test out forms of their satellites.
Angelique Medina:
So if anybody has is in that beta program, let us know what you're seeing and we'll also be covering satellite service conductivity in an upcoming episode. We'll chat a little bit about like some of these new changes and things like SD-WAN and satellite connectivity. So that'll be a fun episode.
Archana Kesavan:
Yep. Looking forward to that. Now, we're going to get under the hood to see what happened with the AT&T and Comcast outage.
Angelique Medina:
Neither of these outages were really ones that a lot of people were complaining about. And we'll talk a little bit about why that is in some cases. So basically, on Wednesday, June 24th, so this was around 8:10 UTC, which in local time in the Seattle, because this took place on the West coast, in the United States. That would have been, I believe 2:10 AM. This was very, very early in the morning, or you can think of it as the middle of the night.
Angelique Medina:
And basically this was really contained to Comcast Network in Seattle. So you can see here that it started just after 2:00 AM, and then it lasted about 25 minutes. And during this period, we could see that there were several sort of interfaces or nodes within the Comcast network—on their backbone and also in some of their business services as well—that were impacted throughout this outage. If we look here, we can see there's some consistency throughout.
Angelique Medina:
And then in some cases there are some nodes that come into play and then go away throughout.
Archana Kesavan:
Yeah, all of them are kind of localized within the Seattle area.
Angelique Medina:
Right. And Seattle is a major kind of peering point as well. So it could have a broad impact presumably. But also, the time of day will make a lot of difference there as well. And then the other one happened the next day and that was the AT&T outage. And this one actually happened in the middle of the day local time. So this took place in Chicago in AT&T's network, and it had an interesting pattern because there were some issues over a small window, and then it just repeated four more times. So started around, let's see, this would have been around ...
Archana Kesavan:
1:00 PM?
Angelique Medina:
2:00 PM in the afternoon?
Archana Kesavan:
Yeah, around 2:00 PM. Yeah.
Angelique Medina:
Central Time. So that's in the middle of the day, it was pretty small. So it wasn't impacting a whole lot of nodes over the course of the outage. But it was also ...
Archana Kesavan:
This kind of outage, it reminds me of that Virgin Media outage on April 29, they had similar patterns repeating every 15 minutes.
Angelique Medina:
Right. So I believe that it was like every hour ...
Archana Kesavan:
Every hour, you’re right.
Angelique Medina:
So that was like very, it was taking place over like many hours, this same pattern. That was really interesting. So it's not clear if this is like some kind of automated thing or some kind of ...
Archana Kesavan:
It was still localized though.
Angelique Medina:
Localized, but it seemed again, it was recurring.
Archana Kesavan:
So Angelique, one of the things that when you're covering these outages stands out is how both the Comcast and the AT&T outage was really localized in an area. So we didn't see like a broad blast radius for the outage. But there's also some differences in terms of when they occurred and what was going on.
Angelique Medina:
So the Comcast outage, even though it was pretty lengthy, so 25 minutes is something where if that were to happen in the middle of the day, is probably something that would get the same attention as like Virgin Media in their outage. Now, granted it wasn't taking place for the whole day, but that's a pretty lengthy outage. But it kind of almost doesn't matter because it's in the middle of the night and that could have been a maintenance window and they could have just been making changes, and then nobody's the wiser, because nobody's really using the Internet at that time. Versus something happening middle of the day, even if it is smaller, it could have a real material impact on users reaching services.
Angelique Medina:
Again, just because it's during like peak usage hours. So not all outages are created equal in some ways. It's not just about size of an outage, but a lot of what it has to do with is when it occurs. Is it occurring when everybody's on holiday or when everybody's asleep? And that will really kind of be the measure.
Archana Kesavan:
Yeah, also apart from just the timing of the Comcast outage, if we just see the pattern, it was consistent over 25 minutes. So it sounds like it was probably a maintenance window that was taking effect.
Angelique Medina:
Yeah. Or some automation or something along those lines that had gone wrong that was contributing to this.
Archana Kesavan:
Yep, yep. That makes sense. I guess that's all we have this week. Don't forget to subscribe where you'll get a free t-shirt by emailing InternetReport@thousandeyes.com with your address and size.
Archana Kesavan:
Also, the registration is open for the State of the Internet. That's coming up the July 16.
Angelique Medina:
July 16. Yeah, we're really excited about that. We have some amazing speakers, we have the Head of Global Content Delivery and Internet Service for CenturyLink. He's going to be talking about how they address some of the challenges they experienced in delivering streaming gaming, really fascinating. It had to do with a number of different things like regulatory agencies in certain regions and things like that, and performance issues. So that's going to be a fascinating talk. And then we also have someone from Fastly, as well. Who's going to be on one of the panels.
Archana Kesavan:
Jana Iyengar from Fastly is going to be on the panel, as well. And I believe he's talking about what they saw in terms of traffic shits with respect to COVID. So that's going to be interesting, as well.
Angelique Medina:
Yeah. And many more speakers coming from ISPs and content delivery providers. So definitely check it out and you can find the registration page for that at thousandeyes.com/events. And feel free to share that on the socials as well.
Archana Kesavan:
Yeah, don't forget to register. And then again, next week we have Atif Khan, CTO at Alkira, who will be here to talk about multi-cloud networking. So be sure to check that out after the 4th of July holiday, for those of you in the US, and we will air that next Tuesday.
Angelique Medina:
So with that, that's the show.