This is the Internet Report, where we uncover what’s working and what’s breaking on the Internet—and why. On this week’s episode, I am joined by Deepak Ravi from our Dublin technical sales engineering team to discuss a recent outage at Garmin. According to its press release, Garmin confirmed that it was a victim of a ransomware attack, which took down several of its services including its website functions, customer support, customer-facing applications, and company communications. In this episode, we walk through what we observed in the ThousandEyes platform during the time of the attack, and what the impacts were on users attempting to access Garmin services. We’re also joined by ThousandEyes’ CISO, Alexander Anoufriev, to talk about what ransomware attacks are, how they manifest and how organizations can protect themselves against future attacks.
Find us on:
Finally, don’t forget to leave a comment here or on Twitter, tagging @ThousandEyes and using the hashtag #TheInternetReport.
Catch up on past episodes of The Internet Report here.
Follow Along with the Transcript
Archana Kesavan:
Hi, and welcome to The Internet Report, where we uncover what's working and what's breaking on the Internet and why. My name is Archana Kesavan, and I'm joined by Deepak Ravi, our technical sales engineer from EMEA. And Deepak's joining us from Dublin today.
Deepak Ravi:
Hey guys. Deepak here. Really excited to talk.
Archana Kesavan:
Thank you, Deepak. So let's get into the headlines really quick. Last week was dominated by two outages. And I think one of these outages got more media attention than the other one. Both occurred on July 23rd.
Archana Kesavan:
The first one was a brief outage in EMEA actually, in London, where broadband providers like BT, TalkTalk, and Plusnet had a meltdown because of a fire in Openreach. Openreach is a subsidiary of British Telecom. But Openreach is responsible for the infrastructure and plumbing that other ISPs rely on for the services. So Openreach basically had a fire in one of their exchanges in Newcastle, and that brought down broadband services. So people were unable to make phone calls from their landlines or browse the Internet.
Archana Kesavan:
The second highlight, and the outage from last week, was the Garmin outage. And that's the one that's been receiving a lot of talk in the media. Because it is an outage that's still not completely resolved, the fifth day in. So the outage started on July 23rd, which was the Thursday. And Garmin is actually still recovering a lot of the services that were impacted, which includes Garmin Connect, Garmin WebMail, garmin.com, and so on. So they are still in the recovery process right now.
Deepak Ravi:
And what's also interesting is although users saw this as an outage for a long period of the time, we slowly started to see news reports that this was probably a cyber attack.
Archana Kesavan:
It was more like a ransomware attack by Evil Corp, which is ... And this is speculation, right? As in, I think there is a little bit of proof now, but none of it has been acknowledged by Garmin at this point.
Deepak Ravi:
That's right. That's right. So we're still yet to get some official confirmation.
Archana Kesavan:
They initially claimed this to be a maintenance window. And then acknowledged that was actually an outage. So, to your point, Deepak, it seems like an outage has been going on for almost four days, 5th day today. But Garmin's not necessarily come out and said anything about why this is happening and what they are doing.
Deepak Ravi:
Exactly. I saw a number of tweets, which actually said, "Hey, why didn't you, as a company, let us know that there is something else going on? Why was maintenance the word being used?" But multiple news organizations actually ended up reporting this as a cyber attack. And that's how the users put the pieces together. So, yeah, I completely agree with you. They could have done a much better job of trying to be more informative, and maybe a little bit transparent, as to what's going on.
Archana Kesavan:
Right. Right. And I think, just in terms of the number of other applications that are relying on Garmin, like Strava was one of them, as well. People were unable to update their workout or download routes and things like that.
Deepak Ravi:
What you mentioned was an example of a regular consumer, fitness and things like that, which is important. But I also understood that Garmin is responsible for official airplane pilot data, that is being downloaded, as well. So, at the end, this is an FAA requirement. I hope I'm getting the abbreviation correctly. But, essentially, it's not just affecting your day-to-day consumers, but actually pilots, as well.
Archana Kesavan:
Yep. Yep. So it's definitely a global and far-impactful outage that, hopefully, Garmin recovers soon, and we hear what they have to say quickly. But, with that, we're going to talk about this Garmin incident a little bit more today. So we're going to go under the hood to see what we saw, from our perspective, on how this unfolded, and some of the takeaways and tidbits we could gain.
Archana Kesavan:
But we're also bringing on into the show, our chief security expert, our CISO from ThousandEyes, who's going to talk a little bit more about ransomware, how to protect your organizations, and what are the best practices you can follow. So stay tuned for the rest of the show, as well. With that, we'll go under the hood.
Archana Kesavan:
All right. So what we're looking at right now is actually WebMail, which is a Garmin service. Again, it's an internal service used by their employees. We're essentially looking at availability of the service. So one of the things that first comes glaring across here is the different availability around July 22nd.
Archana Kesavan:
So this is really interesting, right, Deepak? Because news organizations were calling this outage to have triggered on July 23rd, but we started seeing of this a little ahead. So we saw that. And, as you can see here, we are live here, and it's still not up. So this is, again, one of the services that's still not fully recovered within Garmin.
Archana Kesavan:
But one of the interesting things that we noticed here, apart from this global availability, is just trying to go in here and see a little bit deeper as to what errors we were actually getting. So one of the things that we noticed here that webmail.garmin.com was actually being redirected to adfs.garmin.com here. And that's ADFS. That server was actually what was throwing the internal server error. And this was ... Go ahead.
Deepak Ravi:
And, yeah, I wanted to really pinpoint that specific server. Because this is where we were able to make some educated correlation on whether this could have been a cyber attack. Right? We do understand that cyber attacks, that are ransomware related, tend to attack the active directory services. So maybe there's something to that.
Archana Kesavan:
Right. Also, because I think, I mean, the authentication system, the active directory system is so critical to everything else, that users do or even employees do. So again, there's some correlation, as you said, Deepak, rightly on this being a ransomware attack or a cyber security attack, in this case, and it bringing down ADFS systems.
Archana Kesavan:
But I think what was also interesting is a little further into the timeline. And around 3:00 AM Eastern, so really early in the morning Eastern time, we start seeing a connection reset by peer error. Now this is completely different than what we saw just a few hours ago. But, again, this was an indication that there was no redirect happening at this point in time. Right?
Deepak Ravi:
Yeah, it was failing at the first server that was supposed to be redirecting.
Archana Kesavan:
Right? So this was webmail.com was down, as well, at this point in time. Now, this seemed to be the case throughout the next couple of days. And then sometime around July 25th, at around 8:30 PM Eastern, I believe we started seeing fail to connect to ADFS. We see that redirects have gone up, which means WebMail was now successfully redirecting it to ADFS. But then the active directory system was kind of refusing that connection itself.
Deepak Ravi:
Oh. So, by the way, what's interesting is also that it was only at ... I think I'm kind of jumping timelines here. But it was only at 5:00 AM on July 23rd, that the Garmin team first acknowledged that they were having a problem, which is about six hours since we started detecting a problem on the platform.
Archana Kesavan:
Right, right. We started seeing it at 11:30, July 22nd, and then around 5:00 AM the next day is when Garmin actually ... And this was Garmin India, if I'm not wrong.
Deepak Ravi:
Correct.
Archana Kesavan:
Kind of alluded to there was a problem. Again, all on Twitter, is what we were saying. Right. I think the other interesting observation that we had here was essentially, as we were trying to kind of ... Let me go back to the right timeline that we were looking at, and expanding here a little bit, is we actually were connecting to connect.garmin.com.
Archana Kesavan:
Again, a completely different service than what we were looking at just a little while ago. But we're connecting to Connect Garmin. And this was a service that was down, and affected all the data transfer and synchronization, and all of that across their physical device, and Strava, and all the other apps that we use. Right?
Archana Kesavan:
And, again, from a timeline perspective, around 3:00 AM Eastern, we started seeing an issue. We started noticing that the entire transaction that we had scheduled was failing. And I think it was a “wheel of death,” that was spinning around at that point in time, in terms of logging on to connect.garmin.com.
Deepak Ravi:
Right. And I think it's important to point out that, compared to the previous data set we were looking at, where we were just trying to load the homepage, but here we're actually trying to log in and authenticate with Garmin Connect. And we're seeing the “wheel of death.”
Archana Kesavan:
Right. Right. And then I think the interesting thing, filtering down to identify where exactly the error was happening, we saw 530 response code coming in from Cloudflare now.
Deepak Ravi:
I think, what we first start seeing is a 530 error. And this is exactly the same time when webmail.garmin.com was unable to redirect to adfs.garmin.com.
Archana Kesavan:
Right. I think, interestingly, while it looks like from July 23rd, there was a point in time around July 24th the service actually came up, that's not necessarily what this is indicating. This is indicating that the transcript that we had scheduled to actually log into garmin.com was successfully completing. Because that's the criteria or that's the attribute we are looking at right now.
Archana Kesavan:
But what was interesting here, why was this script actually completing? Were we really able to log into Garmin, is the question. And it turns out we were not. We were not able to log in to Garmin. Instead, we were basically, after the login screen were sent to kind of a standard, "We have an outage going on," just acknowledging that the systems are down. So even though you see this completion percentage here of 100%, doesn't mean Garmin's service was up itself. But you were just being redirected to give you the status.
Archana Kesavan:
Now, in terms of when are these services coming back up, we started noticing around July 27th, at about 14 hours ago, that connect.garmin is slowly up. And this is accurate, in terms of what we're seeing on Garmin's website too, right? You're talking about limited availability of some of their services. And we're also seeing consumers get back, and seeing they're able to sync their workouts and so on.
Archana Kesavan:
And here, if you, again, look at some of the snapshots that you're seeing here, you'll actually be able to see that we're able to successfully go beyond that login platform. And then connect into Garmin and see what your services are, your devices that are set up, and so on.
Deepak Ravi:
Yeah. I think the good news for Garmin here is that they do have some level of delineation between the different services that they host for consumers. And it's kind of good to see that at least some of it's coming back up. It's interesting to note that the WebMail service and the ADFS services haven't still come back up. Right? So we still see those services failing.
Archana Kesavan:
Right. And it makes sense that they're kind of bringing up consumer facing services, and then targeting that, obviously.
Deepak Ravi:
[crosstalk 00:13:31].
Archana Kesavan:
Yeah. Best practices. Again, one of the interesting tweets that I read over the weekend is once Garmin is up, all the sync that has been pending for the last five days, if that's going to trigger ... Is that going to create some kind of data setback? So something that IT teams have to keep in mind, that when you're recovering and bringing back a service, identifying what those dependencies are, and if those dependencies are going to trigger additional load on your systems, kind of take that into consideration before you actually recover, as well.
Deepak Ravi:
Absolutely. Absolutely. I hope they plan for that.
Archana Kesavan:
Yeah. All right. With that, we're going to get into our experts hotline. We have Alexander Anoufriev. Alexander is our Chief Information Security Officer at ThousandEyes. And we thought it would be appropriate to bring him on the call today, to discuss ransomware attacks in general. Because the discussion of this week has been about Garmin's attack/outage, that happened last Thursday on July 23rd. And it's been five days into the incident, and Garmin is still recovering.
Archana Kesavan:
So that would be great to bring Alexander in here, and to just discuss about how ransomware attacks are orchestrated. How do you get out of it? How do you prevent it? And just some interesting topics there. So, Alexander, thanks for being on the show.
Alexander Anoufriev:
Thank you for having me.
Deepak Ravi:
All right. So I think the first line of the curiosity for me, as I was just reading through all the reports that's coming out. Now, again, Garmin has not necessarily claimed or acknowledged that this is ransomware. But there's a lot of press out there, and I think also some evidence, to show that it's ransomware.
Deepak Ravi:
But as I’ve been reading through, there are multiple names that get thrown out. There's Evil Corp, sounds like it's from a movie. Then you have WastedLocker, and so many other malware names that are being put out there. So, for our audience, can you just kind of frame the lay of the land, as to who is Evil Corp, what is WastedLocker, and what is a ransomware attack?
Alexander Anoufriev:
Sure. So it all sounds like comics books. Evil Corp is one comics. WastedLocker is another. WastedLocker is actually a relatively new ransomware family. It's been seen live since May 2020. And it is believed to be operated by so called Evil Corp.
Alexander Anoufriev:
Evil Corp is a group of cyber criminals who has been in business since 2007. And it is originated from Russia. They've developed and deployed a lot of different malware. Their previous ransomware was called Deet Payner. They are also responsible for executing attacks, stealing banking credentials, and many other attacks in the last 13 years.
Alexander Anoufriev:
The name WastedLocker is based on the file extension it adds to encrypted files. And, typically, attacks involving this ransomware hit file servers, database services, virtual machines, cloud environment, and all parts of critical IT infrastructure.
Archana Kesavan:
So the WastedLocker's strain of ransom attacks, is that the part that encrypts? Or does all ransom attack encrypt internal files?
Alexander Anoufriev:
We can think that ransomware has two different flavors. One is traditional low cost execution, around somewhere that is sent to everyone. It's not just custom, it's generic, and starts as a phishing attack. Somebody at home PC executes a malicious file, and then their files get infected. And they're asked to pay $700, for example.
Alexander Anoufriev:
This one, WastedLocker, is highly customized and well planned. It's part of the bigger attack, which is targeted attack. It may take several months to plan and execute. A ransomware attack typically starts either as a phishing email, or vulnerability exploited in a system or application.
Alexander Anoufriev:
In well-planned attacks, when an attacker obtains access to the host, it won't trigger encryption right away. It starts performing analysis. What else can I do? What other damage can I bring as part of this attack? And I spread laterally across all the networks, all the systems.
Alexander Anoufriev:
And, in the end, you may achieve your desired outcome by bringing almost every IT asset down, either by encrypting data of the most critical IT assets, like authentication domain, or virtualization infrastructure, or just putting everything down one by one.
Archana Kesavan:
Wow. Interesting. My next line of question was around how does one orchestrate this attack? And is it just originated or triggered basically through a phishing attack, and then slowly propagates?
Alexander Anoufriev:
Right. It's all of the above. A phishing email, even a social engineering phone call, pretending to be your IT department, and asking to urgently clean up your PC. Combination of techniques.
Alexander Anoufriev:
The owner of an asset that has been compromised is essentially a victim. And they will have a few choices. It's a home-based user, and my files got encrypted, I can either pay $700, as they requested. Or I can just forget about my files and accept the damage. If I have a backup, I can just store files from the backup.
Archana Kesavan:
Something that you mentioned, is about how the attack kind of targets some really critical parts of the infrastructure. Earlier on the show, we were walking through how this attack manifested, from the vantage point of ThousandEyes.
Archana Kesavan:
And one of the things that we noticed in there was, while accessing more testing to an internal domain, a webmail domain of Garmin, we noticed that the ADFS server was ... Either we were unable to reach it, it was doing a 500 error, or it was blatantly just refusing any connection.
Alexander Anoufriev:
Actual directory infrastructure is probably one of the most critical IT assets, and part of enterprise IT infrastructure. It contains user count information, information about computers, service accounts, and other critical information. If it's down, than probably most other critical services will be down as well, databases, web applications. Because they all use authentication credentials.
Alexander Anoufriev:
And employees will not be able to use services, even if they are even independent from the active directory, because their authentication credentials are stored in the directory.
Archana Kesavan:
Right.
Alexander Anoufriev:
And the ADFS, which is an acronym for Active Directory Federation Services, either is a web application, SSO, web SSO. This is how people get out authenticated against internal active directory, to access web SSO applications.
Archana Kesavan:
So in terms of, well, their entire system has been compromised, and all the press and the damage that's happening there. One question that consumers are worried about is around, "Is my password stolen? Is any of my personal information, my credit card information, is that compromised through a ransomware attack?" Is that usually the case, or not?
Alexander Anoufriev:
Yeah. When you talk about this mass cases for ransomware for home PC, you can probably safely say it is not the case. For this one, it is difficult to say. Again, in most cases it's not the case. Making data publicly available or for sale is not the purpose of ransomware attack. The purpose is to get the ransom.
Alexander Anoufriev:
However, of an IT owner or asset owner lost access, or their data, anything may happen. And only a forensic investigation will be able to refine if any data has been exfiltrated or not.
Archana Kesavan:
Got it. Okay. And, in terms of recovery, we're on day five since the incident. And Garmin is coming back up, but still there are some services that I believe are limited as of this morning. Is it normal to see a five day recovery period?
Alexander Anoufriev:
This is probably an indicator of very widespread attack. So the impact is very high, probably maybe the highest we've seen so far. Typically, as part of a recovery activities, you remove all the artifacts left by attacker, all the malicious code, backdoors, compromised accounts that could have been created as part of the attack execution.
Alexander Anoufriev:
And the organization has to rebuild the system. And rebuilding the main controller, or adding to that infrastructure, may take a lot of hours. And then we have to restore the data, perform all the integrity check. When I deal with this, anything else that is required by the incident response process. Also you need to have ...
Archana Kesavan:
Pretty much like. Sorry, go ahead.
Alexander Anoufriev:
Yeah. We also need to understand what has failed, in terms of data protection practices, to make sure to minimize purview to such incident from record. When such a well planned attack is executed, an attacker would also try to disrupt operations, or backup, just the facility. Make sure organization is hurt the most. They don't have running systems, and they can't access backup facility.
Archana Kesavan:
Right, right. Interesting. That's a really interesting point. So, Alexander, towards the end of the conversation here, how would an organization protect themselves from ransomware attacks?
Alexander Anoufriev:
Yeah, there's probably no simple answer to this question, but the answer would lie within the comprehensive security program, that addresses employee awareness indication, physical logical security, business security as well. And let's not forget, backups are really important. Dates are hardly used today, which creates probably an issue of true outside offline backup.
Alexander Anoufriev:
And, as we mentioned previously, if an attacker is able to compromise the primary dataset, they will try to compromise backup dataset as well, or to disrupt restoration facilities.
Archana Kesavan:
Wow. Yeah.
Alexander Anoufriev:
So true offsite backup and true offline backup is really important.
Archana Kesavan:
You're probably going to simulate more testing attacks on us, to see if we are compliant or not.
Alexander Anoufriev:
Yes, we will. You'll continue doing that for sure.
Archana Kesavan:
Well, awesome. Well, thank you so much, Alexander, for being on the call. And walking us through what this is, is 101 on ransomware for us. So thanks again.
Alexander Anoufriev:
Thank you, Archana. Thank you everyone.
Archana Kesavan:
All right. Thank you, Alexander. That was really, really insightful. And I'm hoping we hear back from Garmin exactly as to what happened, and how they were actually planning their recovery, and getting out of this situation as well.
Archana Kesavan:
With that, that's all we have for this week. Don't forget to subscribe, which will get you a free T-shirt, by emailing internetreport@thousandeyes.com, with your address and size. Deepak, thanks again for being on the show, and co-hosting today's episode for our audience.
Deepak Ravi:
Thank you, Archana. It was a pleasure.
Archana Kesavan:
Right. And, again, one more thing before we leave, is we actually have the video for The State of the Internet, is now live on our YouTube channels. If you missed the entire session, or if you just missed parts of it, you can check that out in our YouTube channel. With that, we'll see you next week.