This blog post has been repurposed and reposted from an earlier version originally published on SamKnows.com. Content may have been updated for clarity and relevance.
In this blog post, SamKnows examines the technologies used to deliver live TV streams over the Internet by the BBC and ITV, the U.K.’s two largest terrestrial broadcasters. They also carry out performance measurements from nearly 700 homes in the U.K. during prime time and take a look at how the broadcasters, broadband providers, and the CDNs stack up.
What is live streaming and how is it delivered over the Internet?
Before going any further, let’s be clear about what is meant by “live streaming” and how it differs from “on-demand” or “catch-up” streaming. Live streaming involves watching live TV on the Internet as it is broadcast, often with a small delay, which can be controversial. For example, going to BBC iPlayer and watching the BBC News channel live.
On-demand streaming involves watching pre-recorded programs as and when you feel like it. Watching last week’s episode of your favorite TV show would be a good example of this.
But live TV streaming differs significantly. The “live” aspect makes it significantly more difficult for broadcasters and CDNs — more on this later. On-demand TV providers, such as Netflix, have the luxury of being able to pre-fill their vast array of global caches overnight, ensuring content is always readily available and near to the viewer. This is not so easy with live streams.
There are a variety of mechanisms used to deliver live streams over the Internet. Some broadband providers in the U.K. provide a dedicated TV set-top box, which uses multicast to receive live TV. This is very efficient from a network standpoint, as one stream may be shared by many users concurrently. Some TV providers still use Flash plugins in web browsers, and rely on RTMP (or one of its variants) to deliver content to users. Others use more modern protocols such as HLS, HDS, and MPEG-DASH, which are supported by modern web browsers and, importantly, by Android and Apple mobile devices without the need for plugins. Many TV providers will use a combination of these approaches and will adapt according to the device that the user is using.
Regardless of the protocol, the “content provider” (e.g., the BBC and ITV in this case) will typically not deliver the content directly to end users. Instead, they will rely on an intermediary called a CDN (Content Delivery Network) to handle the vast bandwidth demands for them. In some cases, a content provider may use multiple CDNs. These CDNs will typically have excellent network connectivity to the broadband providers, as they will want to ensure that they can get content to the end users as fast as possible. In some cases, the CDNs will even place “caches” inside ISPs’ networks, which further reduces the distance content has to travel to end users and saves on bandwidth costs for the ISP and CDN.
Live Streaming by the BBC and ITV
The BBC uses two external CDNs called Akamai and Limelight (now known as Edgio), and have more recently built their own CDN called BIDI (“BBC Internet Distribution Infrastructure”). Testing observed only Akamai and Limelight being used for BBC live streaming. The BBC uses a blend of HLS, HDS, and DASH, depending upon the client type.
ITV uses a single CDN — Akamai — to deliver its content to users. It too uses a blend of protocols that depend upon the client type. Desktop web browsers still rely on Adobe Flash plugins, with content being delivered through RTMP, while mobile users are served content using HLS.
Akamai, used by both the BBC and ITV, is one of the largest and oldest CDNs in the world, reportedly serving between 15% and 30% of all web traffic. Their traffic volumes are so large in fact that they offer ISPs caches to install in their networks, which reduces the amount of user requests that need to leave the ISP network. This is done with the goal of improving user experience while simultaneously reducing costs to the ISP, CDN, and content provider. The largest ISPs have all taken Akamai up on this offer and employ Akamai caches inside their networks.
Limelight is a significantly smaller CDN than Akamai. This testing did not observe Limelight caches inside the networks of any of the U.K.’s largest ISPs, meaning that content from Limelight would be served from more distant locations — although, in a country as small as the U.K., this is arguably not significant.
Test Setup
Measurements were conducted on the evening of July 24, 2018, between 6pm and 10pm BST (GMT+1), using 687 SamKnows devices (small, Linux-based appliances) installed in homes across the U.K. These homes had broadband connections provided by BT, Virgin, Sky, TalkTalk, Vodafone, and Hyperoptic. For the VDSL/FTTC (Fiber to the Cabinet) ISPs, testing focused on their fastest products — those with a 76Mbps headline speed. For Virgin, the sole cable ISP, SamKnows only conducted measurements on their 200Mbps and 350Mbps products.
Streams on BBC1 from the BBC and ITV1 from ITV were also tested.
For BBC1, the BBC’s HDS stream was used. The measurement client inside the SamKnows devices connected to the BBC “Media Selector,” which provides links to the manifest files for the different bitrates of the BBC1 streams. Load balancing between Akamai and Limelight occurred at this stage. The client randomly selected between the two CDNs and streamed the highest-quality HD stream of BBC available (target bitrate of 1.7Mbps) for 20 seconds.
For ITV1, ITV's HLS stream, as delivered to mobile apps, was tested. The client fetched the manifest file and downloaded the segments as they were offered to it. The highest quality HD stream available (1.8Mbps) was streamed for 20 seconds, as with the BBC.
The metrics recorded for both the BBC and ITV included:
- Date and time of the transaction
- Hostname and IP address of the CDN server
- Segment ID and size
- Download speed of the segment
- Time to establish the TCP connections
- The “Last Modified” HTTP header of the segment
In total, 16,553 results were collected for the BBC and 7,622 results were collected for ITV.
Analysis
When analyzing the data, the focus was on two particular metrics, the TCP connection time (a rough proxy for round-trip latency), and the download speed (throughput) of the segment.
If any of the following criteria were met, the results were removed:
- The total size of the segment was not more than 0 bytes
- The HTTP response code was not 200
- The TCP connect time exceeded 3000ms, acting as a timeout
The data could be split and aggregated in a variety of different ways to generate metrics, delve deeply into the data, see common patterns, identify anomalies, and produce all the visualizations shown below.
Figure 1 above presents the average download speed from the BBC (blended across Akamai and Limelight) and ITV. At first glance, it might appear that ITV holds a significant edge. However, there are a few things to keep in mind here.
First, the BBC’s average download speed of 37.7Mbps is still far above the actual content bitrate of 1.7Mbps. This would mean that users, on average, would experience no trouble streaming BBC content at this speed. Second, there is some bias towards ITV in the delivery method. The average size of ITV’s segments as delivered through their CDN was 1.4MiB, while the BBC’s was around 800KiB. The smaller file size for the BBC means that TCP is less likely to have time to ramp up to full speed before the 800KiB is fully downloaded.
Figure 2 above presents a scatter plot for the same time range. Performance is quite varied (as would be expected across nearly 700 different broadband connections), but it is consistent — there are no significant drops that would indicate the services are suffering from congestion during peak hours.
Limelight vs Akamai
The fact that the BBC delivers content over both Akamai and Limelight allows for a direct comparison of the two CDNs.
Figure 3 above compares the average download speed for Akamai and Limelight, as well as their average TCP connection establishment time (which is an approximate proxy for round-trip latency). The results show a noticeably higher download speed for Akamai and a lower TCP connection time (lower is better). This is expected given the deployment of Akamai caches inside of the ISP’s networks, which is something that Limelight doesn’t appear to do.
On-net vs Off-net Akamai Performance
Figure 4 shows the presence of Akamai caches inside ISPs networks. Virgin, BT, Vodafone, Sky, and TalkTalk all deployed Akamai caches inside their networks. Of the ISPs measured, only Hyperoptic didn’t have local Akamai caches. That said, there doesn’t appear to be a noticeable penalty for content going “off-net” and being delivered from Akamai’s centralized caches.
Akamai On-net Cache Hit Rate
For an ISP that has invested in deploying Akamai caches inside their network, it’s likely they’ll want to ensure that these caches are being used 100% of the time, and content isn’t being delivered from off-net locations. Figure 5 above shows the volume of requests that were delivered from Akamai caches inside the ISP networks versus those outside of the ISPs’ networks. Curiously, no ISP saw 100% of their requests being delivered from caches inside their networks. BT’s on-net hit rate was 97%, while Vodafone’s was only 78%.
The number of Akamai caches (or at least distinct IP addresses) used in each ISP is also visible. With 105 and 142 distinct IP addresses respectively, BT and Virgin have by far the largest footprint of Akamai caches inside their networks. In reality, these are almost certainly not hundreds of separate servers, but rather multiple IP addresses and/or interfaces attached to a relatively small number of large caching servers.
Limelight on-net caches
Figure 6 above very simply demonstrates that 100% of requests to Limelight were served from Limelight-owned IP addresses. None of the requests were served from the address space of the ISPs, strongly indicating that Limelight caches are not present in the ISPs’ networks.
IPv4 vs IPv6
IPv6 adoption has finally accelerated, and many of the U.K.’s ISPs have caught up, with some already running it in production and others trialing it. However, as Figure 7 shows, neither Akamai's nor Limelight’s CDNs delivered content for the BBC or ITV over IPv6. This isn’t due to the SamKnows devices not having IPv6 connectivity; there are simply no AAAA records associated with the CDN endpoints advertised by the BBC and ITV manifest files. Curiously, the BBC’s manifest files themselves are served over IPv6 and are also served by Akamai, but the content they reference is not.
ISP Comparison
Figures 8 and 9 above plot the average download speed and TCP connection time respectively for each of the ISPs tested to both the BBC and ITV content. Note that the BBC is presented as a blend of Akamai and Limelight delivered content in this chart. While there may appear to be some sizable differences in numbers in these charts, all of them are perfectly adequate for the current video streams available from the BBC and ITV. Again, the relatively small object size (800KiB for the BBC and 1.4MiB for ITV) plays a large role in why speeds are relatively low.
The one interesting outlier is Hyperoptic, which is significantly faster than all other ISPs. While Hyperoptic is the only ISP included with some 1Gbps connections measured, the main reason for their outperformance is the vastly lower latency (which can be seen in the TCP connection time). This allows them to complete the TCP ramp up much more quickly and spend more time at a higher rate when transferring the object, thus increasing the overall average speed.
Conclusion
Streaming live TV over the Internet is a complex business. Content providers like the BBC and ITV make use of an extensive network of invisible infrastructure to ensure that content can be streamed interruption-free. SamKnows’ testing has shown the differences in how two of the major CDNs — Akamai and Limelight — structure their networks to deliver content to end users. Regardless of the approach, performance for both BBC and ITV live streaming services appears to be good across the major U.K. ISPs, at least for the current HD level of video quality.
With the advent of 4K live streaming, which the BBC recently trialed for the World Cup in a limited fashion, bandwidth demands will increase significantly. Reports talk of the BBC using approximately 35Mbps in their 4K streams. Rest assured, SamKnows (now part of Cisco ThousandEyes) will continue developing new measurements to keep track of the performance of ISPs, content providers, and CDNs.