Path Quality Part 1: The Surprising Impact of 1% Packet Loss

It is well understood that packet loss negatively affects network flows. Among network engineers, many understand how the various intricacies of TCP, such as congestion avoidance algorithms, work. TCP questions are actually quite common on network engineering interview loops. But even with all of that, if you were to ask engineers to quantify the effects of packet loss, it is likely you would be met with blank stares. Given how well-researched and widespread packet loss is, it is interesting how little we as a networking community understand its impacts.

Here at ThousandEyes, we often discuss the adverse effects of packet loss, especially in our outage analysis blog series. We often show correlations between the spikes in packet loss, and their negative impacts on the applications users rely on. More often than not, we see that a spike in packet loss overlaps with application degradation.

While operators often see sustained spikes in packet loss, which can overlap with application-level performance degradation, it’s my experience that network engineers tend to look past “small” levels of packet loss (say 1 or 2%). In this blog, I’ll explain why this behavior could be problematic and show results from our own research demonstrating how these “small” packet loss issues can potentially have a big impact on users’ experience.

Various Methods TCP Uses To Handle Packet Loss

TCP uses several methods to manage packet loss events and to recover from them, including Duplicate Acknowledgements, Timeouts, Explicit Congestion Notifications (ECN), Selective Acknowledgements (SACK), and Congestion Avoidance algorithms. TCP senders typically use packet loss and increased round-trip time as key indicators to recognize congestion.

Different congestion algorithms have emerged over the last two decades, tackling the same problem using different approaches. If you look at the systems deployed earlier, it is common to find them using Reno. Today, most operating systems use CUBIC as the default; although, there were significant strides to improve even further, as was done with BBR. Given that we are discussing one of the indicators (packet loss) in this post, it’s important to specify that we tested in the network environment using CUBIC as the congestion algorithm. 

CUBIC: The Default Congestion Avoidance Algorithm

This research is going to provide results for an environment that uses CUBIC as the default congestion avoidance algorithm. In the future, we are planning to revisit this topic with results for other congestion avoidance algorithms. 

Given the increased popularity of the Internet and the growth of computer networks, network engineers quickly realized that some earlier congestion avoidance algorithms, such as Tahoe, utilized available bandwidth slower than they should, especially on higher-bandwidth networks. That prompted further research, and it was discovered that the CUBIC function has properties that could serve congestion avoidance well in high-bandwidth networks. 

Initially, the CUBIC congestion algorithm quickly expands the congestion window (the number of segments that can be sent at once without incurring drops on the path). As it starts approaching the point at which drops previously occurred, it slows down. If no drops occur, then it starts growing slowly at first and then speeds up. This approach proved to work well in high-bandwidth networks. While CUBIC has its own set of challenges, it proved to be more efficient than its predecessors.

Before diving into the results, we wanted to share some details about the setup we used to conduct these measurements.

Test Methodology: Symmetric and Asymmetric Network Paths

We have five Linux hosts configured to forward packets. Each host has a 1 gigabit ethernet interface connected to the switch. Routing is configured statically, as part of which we are pushing traffic from the first host (acting as an iperf3 client) to the last one (which acts as an iperf3 server). To achieve this, we configured subinterfaces in different VLANs, which also required configuration on the switch. The topology we used to conduct the research for a symmetric network path is shown in Figure 1.

The network path in Figure 1 clearly depicts a symmetric network topology. Traffic going in the forward direction will take precisely the same reverse path. This setup closely resembles what network engineers often implement for private networks. They ensure that the networks are symmetrical for the ease of troubleshooting, setting up firewalls, and so on. So, if the issue appears on a node in the forward path, it will show on the same hops in the reverse direction. This eliminates the requirement to run MTR and other classical troubleshooting techniques from both sides—an approach commonly used for troubleshooting asymmetrical network paths, such as those on the Internet.

Since most Internet traffic is asymmetric, we wanted to conduct the same experiments in those conditions. We used the same five Linux hosts and reconfigured them so that the reverse traffic uses a different path (via the use of different VLANs). Figure 2 shows the network topology that we used for this purpose. 

We are using iperf3 to conduct this research. As such, we are interested in throughput as the primary metric. Throughput refers to the amount of data successfully transmitted or received over a network within a specific period. Throughput considers various factors affecting data transmission, such as network congestion, protocol overhead, etc. Unlike bandwidth, which represents the maximum capacity of the channel, throughput reflects the real-world performance and efficiency of the data transmission process.

Establishing a Baseline

To establish the baseline for our testing, we started iperf3 without any packet manipulation. Testing was conducted for three hours (10,800 seconds) and resulted in a mean value of 804.6Mbps for throughput in the symmetric network experiment. Detailed results can be observed in Table 1. 

Table 1. Baseline results for throughput in the symmetric network experiment

We applied the same approach in an asymmetric network test, and we observed a mean value of 864.13Mbps, an increase of 7.3% compared to the values we observed in the symmetric network experiment. Detailed results for throughput in the asymmetric network experiment are shown in Table 2.

Table 2. Baseline results for throughput in the asymmetric network experiment

Measuring Throughput in a Lossy Environment

Once the baseline was established, we introduced the packet loss using the tc ("traffic control") utility. tc is used to introduce traffic control in the Linux kernel, and it has capabilities such as shaping, scheduling, policing, and dropping. tc has an enhancement called netem ("network emulation") that allows adding delay, packet loss, duplication, and other characteristics to packets outgoing from a specific network interface. We used netem to introduce fixed packet loss for each measurement.

The Curious Case of 1% Packet Loss Impacts

We started by introducing 1% packet loss on the Linux device marked H3 on the interface towards device H4.

Symmetric Test Results

In the symmetric network experiment, our testing shows a stark difference between the baseline and the 1% packet loss probing, with 1% packet loss resulting in a mean value of 235.5Mbps of throughput. On average, 1% of packet loss caused a 70.7% decrease in throughput!

Detailed results can be observed in Table 3. Figure 3 shows the plotted baseline throughput and throughput achieved while testing with 1% packet loss in the symmetric network experiment.

Table 3. Throughput results were achieved with a 1% packet loss in the symmetric network experiment

Figure 3 shows a stark difference between baseline measurements, on average, compared to throughput measurements achieved while testing with 1% packet loss in the symmetric network experiment. 

Asymmetric Test Results

In the asymmetric network experiment, testing with 1% packet loss resulted in 222.49Mbps on average. Compared to the average results we achieved while establishing the baseline in an asymmetric network experiment, this accounts for a 74.2% decrease in throughput, an even more significant throughput decrease compared to the symmetric network experiment.

Table 4. Throughput results were achieved with a 1% packet loss in the asymmetric network experiment

Figure 4 shows the difference between throughputs achieved, on average, while establishing the baseline for the asymmetric network experiment and resulting throughput in the environment with 1% packet loss. 

Lastly, comparing the impact of 1% packet loss on throughput in the symmetric and asymmetric networks reveals that the performance of the symmetric network was better in lossy conditions than that of the asymmetric network. We achieved 235.51Mbps testing on the symmetric network, while the same test resulted in 222.49Mbps on the asymmetric network, a 5.5% decrease in throughput on average.

Increasing the Packet Loss

We then increased the packet loss to 2% and measured what happened to the throughput. In the symmetric network experiment, throughput was 175.18Mbps on average, a 78.2% decrease compared to the baseline results achieved in the same network configuration. The throughput we achieved at 2% packet loss represents a 25.6% decrease compared to the test we conducted at 1% packet loss in the same environment. 

Table 5 shows the detailed results. Figure 5 represents a throughput plot, which indicates how big of a difference there was between testing at 2% packet loss and the baseline. 

Table 5. Throughput results were achieved with 2% packet loss in the symmetric network experiment

Throughput at 2% packet loss in the asymmetric network experiment was 168.02Mbps, as shown in Table 6. This represents an 80.5% decrease in throughput compared to the baseline in the same environment. Figure 12 shows the plot indicating the difference between the baseline and throughput at 2% packet loss in the asymmetric network. 

Table 6. Throughput results were achieved with a 2% packet loss in the asymmetric network experiment

Comparing achieved throughputs we achieved in the asymmetric and symmetric networks, the throughput of 168.02Mbps in the asymmetric network represents a 4.2% decrease compared to the throughput achieved in the symmetric network.

Overall Results

Next, we wanted to understand the adverse effects of packet loss up to 10% in both symmetric and asymmetric networks. The results are shown in Tables 7 & 8 and Figures 7 & 8 below.​​

Table 7. Throughput results achieved in the symmetric network while increasing packet loss up to 10%

Table 8. Throughput results achieved in the asymmetric network while increasing packet loss up to 10%

Looking at these results, we can safely conclude that even a “small” amount of packet loss can have catastrophic effects on throughput. In this study, 1% of packet loss caused a 70.7% decrease in throughput. If this represented a customer-facing application, it could result in a poor experience. While we observed a damaging decline in throughput as the percentage of the sustained loss increased, it is evident that the compounding effect of the loss happened fairly early on.

What does this mean in terms of the actual impact on application performance?

Read my colleagues' take on “Understanding the Impact of Packet Loss on Applications” and stay tuned for the results of our next packet loss experiment comparing CUBIC with Reno and BBR.