This blog post has been repurposed and reposted from an earlier version originally published on SamKnows.com. Content may have been updated for clarity and relevance.
Speed tests don't always tell the whole story. For example, a dissatisfied user once reported suboptimal Netflix performance despite having a 100 Mbps connection. Upon investigation, it was discovered that the issue was not isolated to this user. A routing problem was identified, spanning two continents, and Internet service providers (ISPs) were promptly alerted to implement a swift fix.
In another instance, a user in Lithuania reported experiencing frequent Netflix buffering issues, even though they paid for a 100 Mbps broadband connection. Again, the investigation revealed that the problem extended beyond this single user, emphasizing that speed tests alone do not provide a complete picture.
This blog post documents that investigation and its findings.
Background
The user — let’s call him Fred — ran a series of web-based speed tests that consistently showed good results, close to the advertised rate of 100 Mbps. Fred had a SamKnows device installed at home, running automated speed tests and Netflix tests frequently. The device connects to the router and measures only the quality of service provided by the ISP, removing any in-home influences, such as cross-traffic, from the data collected. The Netflix test measures performance to the actual Netflix content servers, using the same CDN redirection logic as the real Netflix application.
Figure 1 above indicates that Fred was achieving good speeds to the nearest dedicated speed test server (located in Riga, Latvia), but his download speeds from Netflix were significantly lower.
Before proceeding, let's review how Netflix delivers video content to users.
How Netflix Delivers Content
Netflix operates a large global content distribution network (CDN) to deliver video content to end users. These CDN caches are referred to as “Open Connect Appliances” (OCAs) in Netflix’s terminology. They may be installed within the ISP’s network if the ISP is large enough and opts to install them, or they may be hosted by Netflix at several major international peering locations.
When a user starts the Netflix application and requests to play a video, an API on Netflix’s side examines the user's source IP address and attempts to determine the most appropriate set of OCAs to serve the user's request. At the very least, this process considers the geographical location of the user and the ISP the user is using. This should ensure that the user’s requests are always served from a nearby Netflix OCA with ample bandwidth; however, this can sometimes fail.
A more detailed description of how Netflix delivers content to end users can be found here.
Finding a Correlation
The first step in investigating a performance issue is to determine whether the issue is observed for just one user or if it’s a wider issue, and if so, what the correlating factor is. In this case, it became clear that the issue was correlated to the ISP Fred uses—Telia Lithuania.
Figure 2 above shows a scatter plot of download speed test results to dedicated speed test servers in Latvia, yielding stable results, while the Netflix download speed results are inconsistent.
Measurements to Netflix, where content is delivered from 45.57.74.0/24, appear to perform optimally, while performance is suboptimal when 45.57.75.0/24 is used. Figure 4 below summarizes this clearly.
Both 45.57.74.0/24 and 45.57.75.0/24 are Netflix prefixes that appear to be announced out of Frankfurt, Germany. Traceroutes from one of the servers in Frankfurt confirmed that the servers were indeed in Frankfurt (round-trip latency was 0.6 ms).
However, this says nothing about the path that traffic from Telia Lithuania takes to these servers. To understand this, some traceroutes were enabled on SamKnows devices for Telia Lithuania to these Netflix addresses.
The observations are shown in Figure 5 below.
From this, it can be gleaned that traffic to 45.57.75.0/24 is taking a completely different path compared to 45.57.74.0/24.
Figure 6 shows that traffic to 45.57.74.0/24 is going directly to Frankfurt and back, resulting in good Netflix performance.
However, Figure 7 shows how traffic to 45.57.75.0/24 is leaving Lithuania, going all the way to Washington D.C. in the U.S.A. (po300.es02.was001.ix.nflxvideo.net), and then back to Frankfurt.
This path adds more than 90 ms compared to using the direct path to Frankfurt. All of this extra distance not only increases latency but also means that the traffic is traversing many more intermediate networks, increasing the likelihood of encountering congestion at some point.
Conclusion
A routing issue has been identified between Telia Lithuania and Netflix’s CDN, causing Netflix traffic for Lithuanian users to sometimes be routed through Washington D.C. This is likely the cause of the underperformance observed on Telia Lithuania broadband connections.
Such performance issues are regularly investigated by ISPs, regulators, and end users. This situation also demonstrates why speed tests to nearby test servers are insufficient to fully characterize the quality of experience.
On a positive note, an offer was made to share these findings and supporting data with Telia Lithuania. Hopefully, this will enable Fred to return to watching Netflix interruption-free in the very near future.