RegisterLogin
Linux Administration

Diagnosing and Resolving eBPF XDP Packet Drops, NIC Ring Buffer Overruns, and SoftIRQ CPU Bottlenecks under High-Throughput UDP/QUIC Traffic

A comprehensive systems-engineering guide to diagnosing kernel packet drops, NIC ring buffer queue exhaustion, eBPF XDP driver-mode drops, and SoftIRQ CPU core saturation under heavy HTTP/3 QUIC and UDP workloads.

Nextgen Hosting Systems Engineer•
Diagnosing and Resolving eBPF XDP Packet Drops, NIC Ring Buffer Overruns, and SoftIRQ CPU Bottlenecks under High-Throughput UDP/QUIC Traffic

Diagnosing and Resolving eBPF XDP Packet Drops, NIC Ring Buffer Overruns, and SoftIRQ CPU Bottlenecks under High-Throughput UDP/QUIC Traffic

As modern web applications migrate from TCP-based HTTP/1.1 and HTTP/2 to HTTP/3 (QUIC), network architectures face a fundamental shift in transport layer behavior. Unlike TCP, which benefits from decades of kernel-space optimizations such as TCP Segmentation Offload (TSO), Generic Segmentation Offload (GSO), and hardware connection tracking in NIC ASICs, QUIC runs over user-space UDP datagrams.

Under sustained ingress traffic—such as millions of concurrent QUIC streams, DNS flood spikes, real-time gaming clusters, or live video ingestion—Linux edge nodes frequently encounter catastrophic packet drop rates and tail-latency degradation.

System engineers commonly observe symptoms where the aggregate network bandwidth is well within physical NIC port capacity (e.g., 2 Gbps on a 10 Gbps / 25 Gbps interface), yet:

  • Application-layer HTTP/3 requests fail with abrupt handshake timeouts or stream reset errors (QUIC_HANDSHAKE_FAILED, ERR_QUIC_PROTOCOL_ERROR).
  • One or two specific CPU cores peg at 100% %si (SoftIRQ) utilization while the remaining cores sit idle.
  • ethtool -S <interface> reports alarming counters for rx_no_buffer_count, rx_missed_errors, or rx_discards.
  • eBPF / XDP (eXpress Data Path) programs deployed for DDoS mitigation or L4 load balancing begin dropping valid ingress packets or aborting with XDP_ABORTED action codes.
top - 14:32:10 up 48 days, 12:44,  1 user,  load average: 14.82, 12.11, 8.45
Tasks: 412 total,   3 running, 409 sleeping,   0 stopped,   0 zombie
%Cpu0  :  1.2 us,  3.4 sy,  0.0 ni, 12.1 id,  0.0 wa,  0.0 hi, 83.3 si,  0.0 st
%Cpu1  :  2.1 us,  2.8 sy,  0.0 ni, 94.8 id,  0.0 wa,  0.0 hi,  0.3 si,  0.0 st
%Cpu2  : 45.0 us, 12.1 sy,  0.0 ni, 42.6 id,  0.0 wa,  0.0 hi,  0.3 si,  0.0 st
%Cpu3  :  0.8 us,  1.2 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi, 98.0 si,  0.0 st

This guide details the end-to-end journey of an ingress UDP/QUIC datagram through the Linux kernel networking stack. We cover diagnostic methodologies using ethtool, bpftrace, perf, dropwatch, and napi_poll telemetry, followed by hardening recipes for NIC ring buffers, Receive Side Scaling (RSS), Receive Packet Steering (RPS/RFS), eBPF XDP memory model allocations, and UDP GRO (Generic Receive Offload) on High-Performance Linux VPS and Dedicated Bare-Metal Clusters.

1. Architectural Anatomy: Linux Ingress Packet Lifecycle

To pinpoint where packets are discarded, we must dissect the ingress pipeline from the physical PCIe PHY layer to user-space application sockets:

+-----------------------------------------------------------------------------------+
|                            Physical Network Interface (NIC)                       |
|   1. Ethernet Frame Arrives -> Transceiver -> MAC/PHY Validation                  |
|   2. DMA Controller copies packet into Ring Buffer (rx_ring) Descriptor           |
|   3. Hardware Interrupt (HardIRQ) asserted to assigned CPU Core                   |
+-----------------------------------------+-----------------------------------------+
                                          |
                                          v
+-----------------------------------------------------------------------------------+
|                        eBPF / XDP (eXpress Data Path) Layer                       |
|   [Native / Driver Mode] Executes directly in NIC driver before sk_buff alloc    |
|   Actions: XDP_DROP | XDP_PASS | XDP_TX | XDP_REDIRECT (AF_XDP / CPU / Dev)      |
+-----------------------------------------+-----------------------------------------+
                                          | (If XDP_PASS)
                                          v
+-----------------------------------------------------------------------------------+
|                           NAPI / SoftIRQ (ksoftirqd)                              |
|   1. HardIRQ disables NIC interrupt and schedules NAPI poll (raise NET_RX_SOFTIRQ) |
|   2. ksoftirqd/napi_poll() drains Ring Buffer descriptors up to netdev_budget     |
|   3. Allocates `sk_buff` (Socket Buffer) metadata container                       |
|   4. Generic Receive Offload (GRO) aggregates UDP datagram chunks                 |
+-----------------------------------------+-----------------------------------------+
                                          |
                                          v
+-----------------------------------------------------------------------------------+
|                        Core Network Stack & Netfilter                             |
|   1. RSS / RPS hashes packet flow -> steers to target CPU backlog (input_pkt_queue)|
|   2. Netfilter Hooks (iptables / nftables / conntrack / tc-bpf)                   |
|   3. Routing Lookup (FIB) -> Transport Layer Demux (UDP `udp_rcv()`)              |
+-----------------------------------------+-----------------------------------------+
                                          |
                                          v
+-----------------------------------------------------------------------------------+
|                             Socket Receive Buffer                                 |
|   1. Enqueued to `sk->sk_receive_queue` (Subject to `rmem_max` / `so_rcvbuf`)     |
|   2. Epoll/io_uring notification -> User space application (`recvmsg` / `recvmmsg`)|
|      (Nginx, LiteSpeed LSPHP, Cloudflare quiche, Envoy, Caddy)                    |
+-----------------------------------------+-----------------------------------------+

The Three Critical Failure Modes:

  1. NIC Descriptor Ring Exhaustion: Incoming packet rate exceeds the rate at which NAPI poller drains descriptors. Hardware drops the packet before the kernel even sees it (rx_dropped, rx_missed_errors).
  2. eBPF XDP Tail & Driver Constraints: In Native XDP mode, memory pool exhaustion (page_pool), driver headroom mismatch, or unhandled return codes result in XDP_ABORTED or unexpected silent drops.
  3. Single-Core SoftIRQ Bottleneck: All ingress interrupts map to CPU0/CPU3 due to poor IRQ affinity or lack of RSS hash entropy for UDP 4-tuples, saturating ksoftirqd and throttling netdev_budget.

2. Low-Level Diagnostics: Isolating the Drop Layer

When packet loss occurs under heavy QUIC or UDP traffic, random sysctl adjustments often exacerbate latency. Follow this systematic diagnostic sequence.

Step 2.1: Inspecting Interface Hardware and Driver Drops

Query the NIC driver counters directly via ethtool. Look for queue-specific buffer overflow counters:

# Display aggregate queue statistics and driver-specific drop counters
ethtool -S eth0 | grep -E -i "drop|miss|err|discard|over|full|budget|no_buf"

Sample output indicating ring buffer overflow:

rx_dropped: 1849204
rx_missed_errors: 1849204
rx_no_buffer_count: 94210
rx_queue_0_drops: 1845112
rx_queue_1_drops: 4092
rx_queue_2_drops: 0
rx_queue_3_drops: 0
rx_out_of_buffer: 94210

[!IMPORTANT] Notice that rx_queue_0_drops accounts for 99.8% of drops. This demonstrates severe RSS queue imbalance: UDP traffic is hashing to queue 0 exclusively, starving that specific ring buffer while queues 2 and 3 remain idle.

Check current vs. maximum ring buffer limits:

ethtool -g eth0
Ring parameters for eth0:
Pre-set maximums:
RX:             4096
RX Mini:        n/a
RX Jumbo:       n/a
TX:             4096
Current hardware settings:
RX:             512
RX Mini:        n/a
RX Jumbo:       n/a
TX:             512

Here, the hardware ring is provisioned with only 512 descriptors, despite supporting 4096. Under bursty UDP traffic, 512 descriptors fill within a fraction of a millisecond.

Step 2.2: Tracking Kernel-Level Drop Points with dropwatch & perf

To confirm whether packets are dropped inside the Linux networking stack (e.g., in udp_rcv, ip_rcv, or sock_queue_rcv_skb):

# Run dropwatch in kernel mode
dropwatch -l kas
Initalizing kallsyms db
dropwatch> start
Scanning at 1.000000s intervals...
1452 drops at location __udp4_lib_rcv+0x6b2
890 drops at location sock_queue_rcv_skb+0xa4
320 drops at location nf_hook_slow+0x7b

Alternatively, trace kernel drop events using perf:

perf record -g -a -e skb:kfree_skb -- sleep 5
perf report --stdio --no-children
# Overhead  Trace output
# ........  ........................................................
#
    62.15%  skb:kfree_skb
            kfree_skb
            __udp4_lib_rcv
            ip_protocol_deliver_rcu
            ip_local_deliver_finish
            ip_local_deliver
            __netif_receive_skb_one_core
            process_backlog
            __napi_poll
            net_rx_action
            __softirqentry_text_start

__udp4_lib_rcv dropping packets indicates that packets successfully traversed NAPI and IP routing, but were discarded because the destination socket’s UDP receive buffer (SO_RCVBUF / rmem) was completely full.

Step 2.3: Tracing eBPF XDP Execution with bpftrace

If an eBPF XDP program is attached (e.g., for WAF filtering, DDoS protection, or Cilium CNI), verify that XDP is not dropping or aborting valid packets due to map lookup failures or bounds-check traps:

# Check if XDP is attached to the interface
ip link show dev eth0 | grep -i xdp

Inspect XDP return actions using bpftrace:

# Attach bpftrace to xdp_exception and xdp_action tracepoints
bpftrace -e '
tracepoint:xdp:xdp_exception {
    @exceptions[args->action, args->errno] = count();
}

tracepoint:xdp:xdp_bulk_tx {
    @bulk_drops[args->drops] = count();
}

interval:s:1 {
    print(@exceptions);
    print(@bulk_drops);
    clear(@exceptions);
    clear(@bulk_drops);
}
'

If @exceptions[XDP_ABORTED, ...] appears, an eBPF program is accessing out-of-bounds packet memory or attempting invalid map lookups, causing the kernel to treat the packet as fatal and drop it immediately.

Step 2.4: Diagnosing SoftIRQ CPU Saturation & NAPI Starvation

Monitor per-CPU SoftIRQ consumption using mpstat:

mpstat -P ALL 1 3
02:15:30 PM  CPU    %usr   %sys   %iowait    %irq   %soft   %idle
02:15:31 PM  all    8.20   4.10      0.00    0.20   22.50   65.00
02:15:31 PM    0    1.00   2.00      0.00    0.00   98.00    0.00   <-- Saturated!
02:15:31 PM    1    9.00   4.00      0.00    0.00    1.00   86.00
02:15:31 PM    2   12.00   5.00      0.00    0.00    0.00   83.00
02:15:31 PM    3    0.00   1.00      0.00    0.00    0.00   99.00

Check /proc/net/softnet_stat to detect NAPI time-slice starvation and backlog queue overflows:

# Print formatted softnet_stat
awk '
{
    printf "CPU%-2d: processed=%-10d dropped(backlog)=%-8d time_squeeze=%-8d cpu_collision=%-6d flow_limit=%-6d\n", 
    NR-1, "0x"$1, "0x"$2, "0x"$3, "0x"$4, "0x"$5
}' /proc/net/softnet_stat
CPU0 : processed=48102911   dropped(backlog)=84190    time_squeeze=120491  cpu_collision=0      flow_limit=0
CPU1 : processed=124010     dropped(backlog)=0        time_squeeze=0       cpu_collision=0      flow_limit=0
CPU2 : processed=98201      dropped(backlog)=0        time_squeeze=0       cpu_collision=0      flow_limit=0
CPU3 : processed=45192      dropped(backlog)=0        time_squeeze=0       cpu_collision=0      flow_limit=0

Understanding the Metrics:

  • dropped(backlog) (Column 2): Number of packets dropped because the per-CPU input_pkt_queue exceeded net.core.netdev_max_backlog.
  • time_squeeze (Column 3): Number of times NAPI’s net_rx_action exhausted its CPU processing time limit (net.core.netdev_budget_usecs) or work quota (net.core.netdev_budget) before draining all packets, leaving unprocessed frames in the ring buffer.

3. Engineering the Solution: High-Throughput Kernel Tuning

Resolving these bottlenecks requires a multi-tiered configuration spanning the NIC hardware layer, kernel networking subsystems, and user-space socket buffers.

+-----------------------------------------------------------------------------------+
|                            MULTI-LAYER HARDENING ARCHITECTURE                     |
+-----------------------------------------------------------------------------------+
| 1. NIC Ring Buffers   | ethtool -G eth0 rx 4096 tx 4096                           |
| 2. Interrupt Affinity | Set irqbalance ban list / Bind IRQ vectors to NUMA cores   |
| 3. Multi-Queue RSS    | Symmetric Toeplitz hash for UDP 4-tuples (ethtool -N)     |
| 4. Software RPS / RFS | Distribute single-flow SoftIRQ across all available cores  |
| 5. NAPI Poller Budget | net.core.netdev_budget = 600, netdev_budget_usecs = 8000  |
| 6. UDP Memory Limits  | net.core.rmem_max = 67108864, rmem_default = 33554432     |
| 7. UDP GRO & GSO      | ethtool -K eth0 rx-gro-hw on, gro on                      |
+-----------------------------------------------------------------------------------+

Step 3.1: Expand Hardware NIC Ring Buffers

Immediately max out the RX/TX descriptor rings to absorb bursty UDP spikes:

# Check maximum supported descriptor size
ethtool -g eth0

# Apply maximum ring buffer capacity
ethtool -G eth0 rx 4096 tx 4096

To persist this setting across reboots in Debian/Ubuntu (/etc/network/interfaces or systemd-networkd) or RHEL/Rocky Linux (/etc/sysconfig/network-scripts):

Create /etc/systemd/system/nic-ring-tuning.service:

[Unit]
Description=Optimize Network Ring Buffers for High-Throughput QUIC
After=network.target

[Service]
Type=oneshot
ExecStart=/usr/sbin/ethtool -G eth0 rx 4096 tx 4096
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

Enable the service:

systemctl daemon-reload
systemctl enable --now nic-ring-tuning.service

Step 3.2: Configure Multi-Queue RSS & Symmetric Hashing for UDP

Standard NIC firmware hashes TCP 4-tuples (src_ip, dst_ip, src_port, dst_port) across hardware queues, but defaults to hashing only the 2-tuple IP header (src_ip, dst_ip) for UDP traffic to prevent out-of-order packet delivery. For high-volume HTTP/3 QUIC, this concentrates all traffic from a single client subnet onto one RX queue.

Enable full 4-tuple hashing for UDP:

# Enable 4-tuple flow hashing for IPv4 UDP
ethtool -N eth0 rx-flow-hash udp4 sdfn

# Enable 4-tuple flow hashing for IPv6 UDP
ethtool -N eth0 rx-flow-hash udp6 sdfn

Verify the hash configuration:

ethtool -n eth0 rx-flow-hash udp4
UDP over IPv4 flows use these fields for computing Hash flow key:
IP SA
IP DA
L4 bytes 0 & 1 [TCP/UDP src port]
L4 bytes 2 & 3 [TCP/UDP dst port]

Step 3.3: Implement Receive Packet Steering (RPS) and Flow Steering (RFS)

If your NIC has fewer hardware queues than physical CPU cores (common in cloud instances and virtualization), configure RPS to distribute SoftIRQ packet processing across all CPU cores.

For a 4-core server (bitmask f = binary 1111):

# Distribute queue 0 packet processing across CPU cores 0-3
echo "f" > /sys/class/net/eth0/queues/rx-0/rps_cpus

# For multi-queue NICs, assign dedicated CPU masks to each queue
for q in /sys/class/net/eth0/queues/rx-*; do
    echo "f" > "$q/rps_cpus"
done

Enable Receive Flow Steering (RFS) to direct packet processing to the CPU core where the application socket thread is running:

# Allocate global RFS table size (power of 2)
sysctl -w net.core.rps_sock_flow_entries=32768

# Allocate per-queue RFS flow count
for f in /sys/class/net/eth0/queues/rx-*/rps_flow_cnt; do
    echo 8192 > "$f"
done

Step 3.4: Kernel Sysctl Networking Parameter Optimization

Apply a production-grade sysctl profile designed for high-concurrency UDP, HTTP/3, and large NAPI budgets.

Edit /etc/sysctl.d/99-high-throughput-udp.conf:

# ====================================================================
# High-Throughput UDP & HTTP/3 QUIC Kernel Optimization
# Optimized for Nextgen High-Performance Edge Nodes
# ====================================================================

# Maximum number of packets queued in the per-CPU input backlog before dropping
net.core.netdev_max_backlog = 100000

# Number of packets drained by NAPI in a single polling cycle
net.core.netdev_budget = 600

# Maximum microseconds NAPI poller can spend processing a softirq batch
net.core.netdev_budget_usecs = 8000

# Maximum and default socket receive buffer (64MB max, 16MB default)
net.core.rmem_max = 67108864
net.core.rmem_default = 16777216

# Maximum and default socket send buffer (64MB max, 16MB default)
net.core.wmem_max = 67108864
net.core.wmem_default = 16777216

# UDP memory pressure thresholds (min, pressure, max in 4KB memory pages)
# 4GB max memory allocation for UDP subsystem
net.ipv4.udp_mem = 262144 524288 1048576

# UDP socket receive/send buffer tuning (min, default, max in bytes)
net.ipv4.udp_rmem_min = 16384
net.ipv4.udp_wmem_min = 16384

# Enable UDP Early Demux for faster route lookup
net.ipv4.udp_early_demux = 1

# Increase system-wide file descriptor and connection limits
fs.file-max = 2097152
net.core.somaxconn = 65535

# Optimize connection tracking hash tables if netfilter is enabled
net.netfilter.nf_conntrack_max = 1048576
net.netfilter.nf_conntrack_udp_timeout = 30
net.netfilter.nf_conntrack_udp_timeout_stream = 120

Apply the configuration immediately:

sysctl -p /etc/sysctl.d/99-high-throughput-udp.conf

Step 3.5: Enable UDP Generic Receive Offload (GRO)

UDP GRO aggregates multiple ingress UDP packets belonging to the same flow into a single sk_buff before passing it up the network stack. This reduces SoftIRQ overhead by up to 65%:

# Enable GRO on the interface
ethtool -K eth0 rx-gro-list off gro on

For applications utilizing user-space QUIC implementations (such as LiteSpeed LSPHP, Cloudflare NGINX, or Envoy), verify that UDP Segment Offload (GSO) and GRO are active:

ethtool -k eth0 | grep -E "generic-receive-offload|generic-segmentation-offload|udp-fragmentation-offload"
generic-segmentation-offload: on
generic-receive-offload: on
udp-fragmentation-offload: off [fixed]

Step 3.6: Hardening eBPF XDP Memory Allocations

When using eBPF/XDP programs for ingress packet filtering (e.g., custom XDP firewalls or Cilium), ensure that the NIC driver’s page pool is sufficiently sized and does not leak packet frames:

  1. Verify XDP Program Flags: When attaching XDP, prefer xdpdrv (Native Driver Mode) over xdpgeneric (Generic Mode). Generic XDP allocates sk_buff before executing the BPF bytecode, eliminating the zero-copy advantage.
# Attach in Native Driver Mode
ip link set dev eth0 xdpdrv obj xdp_filter.o sec .text
  1. Handle Tail Calls and Fallbacks: Ensure all non-matching branches explicitly return XDP_PASS rather than defaulting to 0 (XDP_ABORTED):
// Example eBPF XDP Ingress Filter Snippet
#include <linux/bpf.h>
#include <bpf/bpf_helpers.h>
#include <linux/if_ether.h>
#include <linux/ip.h>
#include <linux/udp.h>

SEC("xdp")
int xdp_quic_filter(struct xdp_md *ctx) {
    void *data_end = (void *)(long)ctx->data_end;
    void *data = (void *)(long)ctx->data;

    struct ethhdr *eth = data;
    if ((void *)(eth + 1) > data_end)
        return XDP_PASS; // Bounds check safety

    if (eth->h_proto != __constant_htons(ETH_P_IP))
        return XDP_PASS;

    struct iphdr *ip = (void *)(eth + 1);
    if ((void *)(ip + 1) > data_end)
        return XDP_PASS;

    if (ip->protocol == IPPROTO_UDP) {
        struct udphdr *udp = (void *)(ip + 1);
        if ((void *)(udp + 1) > data_end)
            return XDP_PASS;

        // Allow HTTP/3 (port 443) and standard DNS (port 53)
        if (udp->dest == __constant_htons(443) || udp->dest == __constant_htons(53)) {
            return XDP_PASS;
        }
    }

    return XDP_PASS;
}

char _license[] SEC("license") = "GPL";

4. Verification and Benchmark Telemetry

After applying the optimizations, validate network performance under synthetic or real-world high-throughput UDP load.

Verification 1: Re-inspect softnet_stat Under Load

Generate high-throughput UDP/QUIC test traffic and monitor /proc/net/softnet_stat:

awk '
{
    printf "CPU%-2d: processed=%-10d dropped(backlog)=%-8d time_squeeze=%-8d\n", 
    NR-1, "0x"$1, "0x"$2, "0x"$3
}' /proc/net/softnet_stat
CPU0 : processed=89410214   dropped(backlog)=0        time_squeeze=0
CPU1 : processed=88120440   dropped(backlog)=0        time_squeeze=0
CPU2 : processed=89012301   dropped(backlog)=0        time_squeeze=0
CPU3 : processed=87941022   dropped(backlog)=0        time_squeeze=0

Notice that packet processing is now identically distributed across all four CPU cores, with zero backlog drops and zero time_squeeze stalls.

Verification 2: Check Interface Hardware Counters

Confirm that ethtool drop counters remain stationary:

ethtool -S eth0 | grep -E -i "drop|miss|err|no_buf"
rx_dropped: 0
rx_missed_errors: 0
rx_no_buffer_count: 0
rx_queue_0_drops: 0
rx_queue_1_drops: 0
rx_queue_2_drops: 0
rx_queue_3_drops: 0

5. Summary Reference: Production Sysctl & Driver Matrix

Subsystem Component Default Value Production Optimized Value Rationale
NIC Ring Buffer (RX) 512 4096 Prevents hardware descriptor ring drops during microbursts.
net.core.netdev_max_backlog 1000 100000 Prevents per-CPU backlog drops during SoftIRQ transitions.
net.core.netdev_budget 300 600 Allows NAPI to process more packets per softirq execution.
net.core.netdev_budget_usecs 2000 8000 Extends polling time slice under sustained 10G/25G load.
net.core.rmem_max 212992 67108864 (64 MB) Allows application UDP socket buffers to absorb QUIC bursts.
rx-flow-hash udp4 sd (IP only) sdfn (IP + Ports) Distributes multi-stream QUIC evenly across RSS queues.
ethtool gro on / off on Aggregates UDP chunks, slashing per-packet SoftIRQ CPU load.

6. Next Steps & Infrastructure Recommendations

When managing mission-critical web applications handling millions of HTTP/3 QUIC sessions, underlying virtualization and kernel architectures dictate real-world reliability:

  1. Bare-Metal & KVM Virtualization: Ensure your hosting environment provides direct hardware virtualization features (SR-IOV, dedicated vCPUs, and virtio-net multi-queue support). Explore Nextgen High-Performance Linux VPS for pre-optimized kernel networking templates.
  2. Dedicated Network Infrastructure: For enterprise-scale streaming, API gateways, or financial trading platforms demanding multi-gigabit UDP throughput without noisy-neighbor packet jitter, deploy Nextgen Dedicated Bare-Metal Servers.
  3. Web Server HTTP/3 Troubleshooting: For web server specific configurations (Nginx QUIC module, LiteSpeed LSPHP, and Brotli compression), see our guide on Troubleshooting HTTP/3 QUIC Connection Drops in Nginx and LiteSpeed.
  4. Firewall & Conntrack Tuning: To prevent state table exhaustion when handling high-concurrency UDP connections, consult Troubleshooting Linux Conntrack Table Exhaustion and eBPF Firewalls.