DNS-over-HTTPS Performance

This blog post has been repurposed and reposted from an earlier version originally published on SamKnows.com. Content may have been updated for clarity and relevance.

While the vast majority of web browsing traffic (HTTP) is encrypted nowadays, DNS traffic is almost never encrypted. This means that DNS is still susceptible to surveillance and manipulation.

DNS-over-HTTPS was proposed at the IETF in 2018 as a remedy for this. As the name suggests, this effectively wraps DNS traffic inside HTTPS traffic, meaning that the DNS requests and responses cannot be intercepted in clear text or manipulated.

The DNS-over-HTTPS proposal has gained traction among browser vendors.

However, the DNS-over-HTTPS (DoH) resolvers that browsers use are not provided by the ISPs. This, combined with the possibility of browsers enabling DoH by default and using third-party DNS servers, raises a major concern for ISPs.

Arguments for DoH

The main arguments in favor of browsers adopting DNS-over-HTTPS (DoH) are simple and obvious:

1. Improved Privacy: It improves privacy by preventing interception. While most web traffic is encrypted nowadays, you can still tell a lot from DNS requests alone. This can reveal not only what websites you are looking at, but also what devices you have in your home. Of course, whoever operates your DoH resolver would have full visibility of your DNS requests.

2. Prevention of Hijacking: It prevents the hijacking of your DNS traffic, and the manipulation of requests and/or responses.

There is a more subtle benefit to DoH that was discovered in a paper titled Comparing the Effects of DNS, DoT, and DoH on Web Performance: DNS-over-HTTPS operates over TCP, which can retransmit data very quickly in the case of packet losses, whereas traditional DNS clients use UDP and wait for a fixed time before retrying. So, in lossy networks, DoH may outperform UDP-based DNS. However, DoH-over-TCP has head-of-line blocking (i.e., if there is significant packet loss then all requests over that TCP connection slow down). DoH-over-HTTP/3—which uses QUIC under the hood, which in turn uses UDP—would go a long way to resolving this.

Arguments Against DoH

ISPs have been the primary opponents of the current DoH deployment plans. The main objection is that the use of Google/Cloudflare as the default DoH resolvers means that the ISP's DNS servers will no longer be in the path of their customers’ DNS requests. This, in turn, means:

1. Lack of Visibility: The ISP will not be able to see customer DNS traffic. Visibility of DNS traffic allows many ISPs to perform lawful interception and content filtering (either to comply with legal requirements or to offer their own parental control products).

2. Potential Performance Issues: Performance may be worse for their customers, as their customer DNS requests will now have to travel further to receive a reply (ISP's DNS servers typically operate inside the ISP network and are, therefore, closer to the customer than Google/Cloudflare's DNS servers).

3. Impact on CDN Relationships: It will impact the ISP's CDN relationships. Some CDNs rely on DNS to steer traffic to CDN cache nodes operating inside ISP networks. Akamai is the largest example of a CDN that uses this approach. YouTube and Netflix steer traffic towards their caches using application layer logic rather than relying on DNS, so they are unaffected by the DNS provider in use.

It is important to note that the “default-on” deployment of DoH by Chrome and Firefox has not happened yet. It is still a hotly discussed topic. Both browser vendors have indicated that they will not switch it on by default in its current form, in response to the feedback from ISPs.

Measuring Performance

Putting aside the privacy topic, SamKnows (now part of Cisco ThousandEyes) can measure the impact on performance from using DoH.

This study will compare UDP-based DNS (referred to as 'Do53') and DoH across multiple DNS providers, focusing on the following performance aspects:

• DNS resolution time for an Akamai hostname
• The ratio of responses that steer users towards an Akamai cache inside the ISP's data centers and IP space ('on-prem')
• The round-trip time to the resolved IP address of the Akamai hostname
• The web page loading time, including time-to-first-visual and time-to-visually-complete

Comparisons will be made using the following DNS providers:

• ISP-provided Do53
• Google Do53
• Cloudflare Do53
• Google DoH
• Cloudflare DoH

The Akamai hostname used in all DNS measurements is a2.w10.akamai.net.

Akamai operates a very large CDN, with caches operating at many layers of the network, and often works closely with ISPs to cache content as near to the user as possible. In their deepest deployments with ISPs, Akamai has caches installed inside ISP data centers and shares their IP space. These 'on-prem' caches benefit everyone: the customer experiences faster content delivery, the ISP benefits from reduced transit/peering usage, and Akamai can reduce the demands on their centralized infrastructure.

When querying for the hostname a2.w10.akamai.net, you are returned an IP address of the nearest Akamai cache. This may even be an IP address that is within the ISP's IP address space. This Akamai hostname is the one that underpins many large video-on-demand services, including BBC iPlayer (for example, iPlayer uses vod-dash-uk-live.akamaized.net, which is a CNAME to a2.w10.akamai.net). So, if an ISP has Akamai caches locally, a2.w10.akamai.net is expected to resolve to an IP address belonging to that ISP's IP ranges. While testing this hostname was done extensively across Europe, its suitability was not assessed for use in other parts of the world.

It is important to acknowledge the following limitations in this work:

• The TLS setup time is intentionally not included in the DoH results. Firefox typically uses the same TLS session for multiple DoH requests, so the initial TLS connection overhead is amortized across multiple requests.

• 'ISP-provided Do53' is categorized as whatever is returned to the SamKnows device through DHCP. It is acknowledged that some users may have reconfigured their CPE to use a different upstream DNS server; ISPs report that this is typically <0.1 percent of their customer base.

• The performance of DNS-over-TLS (DoT) is not examined.

• Measurements are taken from a very small sample of devices—typically between 2 and 40 SamKnows devices per ISP represented here. Given that DNS services are centralized and the focus is not on comparing ISPs against one another (instead, the interest is in their interactions with DNS providers), this is not a significant concern.

• The results presented are not intended to be used to compare ISPs against one another—the sample size is far too small. They are intended to provide some comparison between the DNS providers and protocols.

• Only performance to Akamai's CDN is examined. This has been chosen because it's a very large CDN, with deep ISP penetration, and its traffic distribution mechanism relies heavily on DNS. A fuller study would look at a broader set of CDNs, including Level3, Limelight (now known as Edgio), StackPath, and others.

Name Resolution Performance

The first test looks at the average DNS resolution time for the Akamai hostname, split by DNS provider, protocol, and ISP. Table 1 below summarizes the results.

Table 1. DNS resolution times for a2.w10.akamai.net, split by DNS provider. All values in milliseconds. Lower is better. Sample size of 299922 tests across 400 SamKnows devices.

The first thing observed in the results is that there is no significant performance difference (benefit or harm) in using DoH versus Do53. The difference between Do53 and DoH on the same DNS provider is typically only a few milliseconds—certainly within the error margins afforded by this small sample.

Much larger differences were seen when comparing DNS providers against one another. It might be surprising to see that Cloudflare delivers DNS resolution time on par with the ISP's own DNS servers in most cases, with Cloudflare returning averages of 23.4 ms for Do53 and 25.2ms for DoH, while ISP Do53 servers averaged 25.6 ms. Google delivers the slowest DNS resolution time in almost all cases (48.8 ms for Do53 and 51.4 ms for DoH).

One likely reason for Cloudflare's strong performance in this metric is that Cloudflare does not support EDNS Client Subnet. This means that Cloudflare can much more aggressively cache DNS responses as their cache does not need to be scoped per subnet.

In Europe, a highly developed Internet market, it should not be surprising that all of the DNS providers have broad coverage and can generally deliver low DNS resolution times.

Use of On-prem Akamai Caches

The second test looks at how frequently the various DNS providers will steer responses towards an Akamai cache located within the ISP's data centers and IP space. These 'on-prem' caches rely on the resolved IP address being within the ISP's IP address space to determine their on-prem status. These results are shown below in Table 2.

Table 2. The percentage of responses for a query for a2.w10.akamai.net that were steered to an on-prem Akamai cache. Higher is better (indicates a higher cache utilization ratio). Sample size of 299922 tests across 400 SamKnows devices.

Before comparing different DNS providers, it’s important to note that rows have been removed where it was unclear what deployment model Akamai used with the ISP. Akamai will sometimes deploy servers inside ISP data centers and use their IP space (termed 'on-prem' for the purposes of this post), and this is very easy for us to identify. In other cases, they may deploy servers in ISP data centers but still use Akamai IP space. In other cases still, they may colocate servers themselves in a neutral data center and use their own IP space, but still have private peering with the ISPs. Without a clear way to distinguish between these last two scenarios, the focus is only on the first scenario (which can be clearly distinguished based upon the resolved IP address).

As can be expected, the vast majority of the ISPs sampled clearly had Akamai caches present on-prem. Deutsche Telekom, Telecom Italia, and Vodafone Italy have been removed as they fell into one of the ambiguous cases discussed above.

For ISPs that do clearly have Akamai caches on-prem, it can be observed that users' DNS responses are reliably steered towards these caches when using the ISP's Do53 servers, Google Do53, and Google DoH. This demonstrates that Google's DoH resolver does honor EDNS Client Subnet. Moreover, there is little difference in on-prem cache hit rate between ISP resolvers, Google Do53, and Google DoH (with a couple of exceptions).

The downside to Cloudflare's decision to not support EDNS Client Subnet can be seen very clearly in the results above. Cloudflare DoH did not steer a single response to an Akamai on-prem cache, for any ISP. Strangely, Cloudflare Do53 appears to behave differently from Cloudflare DoH for a few ISPs—Sky and Vodafone in the UK, and Vodafone in Spain. These three ISPs report a sizable number of responses from Cloudflare Do53 being sent to on-prem Akamai caches, despite Cloudflare not supporting EDNS Client Subnet. How is this possible? The cause of this is likely the hijacking of DNS requests, which are redirected to the ISP's own DNS servers. This is possible on Do53, precisely because the traffic is unencrypted.

It is also possible to observe a few ISP-specific behaviors from the table above that would be of interest to network engineers in those ISPs:

• BT (UK): While almost 100 percent of DNS requests made to BT's own Do53 DNS servers are steered to on-prem Akamai caches, only 80 percent of DNS requests made to Google's Do53 and DoH servers are steered to an on-prem Akamai cache. There's an extra 20 percent free cache hit rate available for BT here.

• Plusnet (UK): Plusnet is a subsidiary of BT, and they make use of the Akamai caches within BT's network. However, only 68 percent of DNS requests to Plusnet's Do53 servers are steered to the on-prem Akamai caches, and only 7 percent of those made to Google's Do53 and DoH servers use the on-prem caches.

• Sky (UK), Vodafone (UK and Spain): These ISPs appear to be hijacking Do53 requests to Google and Cloudflare and are redirecting them to their own ISP Do53 servers.

Round-trip Latency to Resolved IP Address

The third test looks at the round-trip latency to the resolved IP address from each of the DNS providers. No filtering was performed to determine whether the Akamai cache is on-prem or not; the results are simply blended. These results are shown below in Table 3.

Table 3. Round-trip latency to the resolved IP for a2.w10.akamai.net using different DNS providers. All values in milliseconds. Lower is better. Sample size of 271660 tests across 400 SamKnows devices.

There is almost no difference between round-trip latency for IP addresses resolved over Do53 and DoH for a single DNS provider. As with previous tables, the differences are far more closely tied to the DNS providers themselves, rather than the resolution mechanism.

The impact of Cloudflare's decision to not support EDNS Client Subnet is very visible in this table, too. Cloudflare consistently returned IP addresses with higher round-trip latencies than other DNS providers. Moreover, for 9 of the 20 ISPs measured, there was at least a 50 percent increase in RTT over the best case when using Cloudflare. The most severe cases were found in Greece, where round-trip latency to the Akamai hostname increased from 17-31 ms to 104-110 ms when using Cloudflare's resolvers.

Web Page Loading Time

Lastly, the focus is on web page loading time when using DoH versus ISP-provided Do53. This last test was only performed on SamKnows devices in the UK and did not use Do53 from Google or Cloudflare (only DoH was used from these providers). The page https://www.bbc.co.uk/news/ was used as the target web page, which is a very popular destination in the UK.

While the BBC uses Akamai extensively, none of the assets on the BBC News homepage were served from on-prem Akamai caches within ISP networks at the time of testing (the on-prem caches are certainly used for other BBC services, though, such as videos on iPlayer).

Table 4 shows the results of this test.

Table 4. Web page rendering time, split by DNS provider. All values in seconds. Lower is better.

The results here show that there is minimal difference in accessing BBC News when using ISP-provided Do53, Google DoH, and Cloudflare DoH. The time-to-first-visual varied by at most 10 percent, with most ISPs seeing differences of less than 3 percent. The time-to-visually-complete varied by at most 8 percent. There are no stand-out issues to see here.

Of course, if a web page that experienced ~100 ms increases in RTT, similar to what was observed in Greece with Cloudflare, was tested instead, then that would likely have a much more noticeable impact on web page loading time.

Conclusion

In this blog post, a range of performance topics related to DNS-over-HTTPS have been examined.

A common thread among all of these findings was that there was no tangible performance difference between Do53 and DoH for a single DNS provider (e.g., Google Do53 performed extremely similarly to Google DoH in all cases). However, much larger differences were found between the DNS providers themselves (regardless of whether Do53 or DoH were used).

It was found that ISP-provided DNS servers and Cloudflare's DNS servers (both DoH and Do53) delivered the fastest DNS resolution times. Google's were noticeably slower. This may be due to the fact that Google honors EDNS Client Subnet, whereas Cloudflare does not, allowing Cloudflare to serve more responses from its cache.

The study also examined the percentage of DNS responses for an Akamai hostname that steered clients towards Akamai cache nodes located inside ISP data centers and IP space ('on-prem'). It was found that the majority of ISPs tested had Akamai caches on-prem. Using ISP-provided Do53 servers, Google Do53 servers, or Google DoH resulted in a high hit rate for these on-prem caches. However, using Cloudflare's Do53 or DoH servers resulted in almost no traffic being directed to the on-prem Akamai caches. This is not a fault or limitation of DoH in any way; this is caused by Cloudflare's decision not to support EDNS Client Subnet. Additionally, cases were found of some ISPs hijacking DNS queries to redirect them to their own servers, which works for Do53 but not DoH.

Perhaps most importantly, the round-trip latency to the IP addresses returned by the various DNS providers was examined. Again, no performance benefit or harm was found in using DoH versus Do53. Significant differences between DNS providers were identified, though. In some instances, using Cloudflare's Do53 or DoH resolvers increased round-trip latency to Akamai very significantly (by over 100 ms).

Lastly, the web page loading time of BBC News for ISPs in the UK was examined. No meaningful difference was found between using ISP-provided Do53, Google DoH, and Cloudflare DoH resolvers in this scenario.

In summary, no significant performance delta was found between Do53 and DoH. However, significant performance differences were found between the DNS providers themselves, which were visible regardless of the resolution mechanism in use.

Purely from a performance perspective, it might be interesting to consider what would happen if Chrome and Firefox enabled DoH by default tomorrow. If Chrome did this and used Google's DoH servers, then the results demonstrate that there would likely be minimal performance difference to the user, and on-prem Akamai cache utilization would not be materially affected. However, if Firefox enabled DoH by default tomorrow and used Cloudflare's DoH servers, then some users might experience increased latency to Akamai's CDN, and Akamai's on-prem caches would effectively disappear. This would mean that more traffic would need to go to Akamai's off-net caches, thus increasing pressure on ISP peering/transit links.