Ding-Dong Ditch: The Doorbell Prank in FreeBSD's iflib

Since 2017, FreeBSD®'s iflib(4) has been playing ding-dong ditch on network interfaces. When the transmit queue was nearly empty, iflib pressed the interface's doorbell three times for every packet, and two of those announced nothing new. The interface still had to answer each one, and answering is slow. On a busy router, the cores spent most of their time on the doorstep instead of forwarding packets.

We found it on a Netgate® 4200, a small four-core router with 2.5 gigabit Ethernet ports. Flooded with the smallest packets, it topped out at 0.96 million packets per second (Mpps) in one direction, about a quarter of line rate. A one-line fix took it to 2.57 Mpps, and a second change, which presses the doorbell once per batch of packets, took it to 3.58 Mpps. That's within 4 % of the 3.72 Mpps the wire can carry.

How a router sends a packet

To send a packet, the operating system writes a short note about it (where it is in memory and how long it is) into a list the network interface reads from. The list has a fixed number of slots that are reused in a circle, so it's called the transmit ring. The operating system then writes a number into one of the interface's registers to say the list has grown. That write is the doorbell press.

A doorbell write is expensive. The register lives on the interface, not in ordinary memory, so each write costs the processor hundreds of cycles, and the interface can only accept so many writes per second.

On FreeBSD, most Ethernet drivers share this machinery through iflib, a library that manages the transmit and receive rings. Each driver supplies only the hardware-specific parts, such as how to write to its doorbell register. A bug in iflib therefore reaches every driver built on it, including igc(4), the driver for the 4200's Intel 2.5 GbE ports, which Netgate sponsored and adapted to iflib.

A router stuck at a quarter speed

To measure how fast the 4200 forwards, we sent it a steady stream of 64-byte frames (the smallest Ethernet allows), raised the rate step by step, and counted what came out the other side. Small packets are the hardest test, because the work per packet stays the same while the packets arrive faster.

With the router set to four queues per port, spreading the work over all four cores, forwarding stopped at 0.96 Mpps in one direction (unidirectional). Between 0.8 and 1.0 Mpps the cores went from 36 % busy to fully busy, far more than 25 % more traffic should cost. That ruled out the obvious explanation. If the router were simply doing too much work per packet then turning on the pf firewall, which adds work to every packet, would have lowered the ceiling. It didn't. Raising iflib's receive budget, the number of packets the receive task takes from the receive ring in one pass, from its default of 16 to 64 and then 256 didn't raise it either. A profile of the cores pointed at a single function: the one that writes the doorbell.

Three doorbell writes per packet

iflib is meant to batch its doorbell writes. Before writing, it counts the packets waiting to be announced and holds off until there are enough. How many is enough depends on how full the transmit ring is. When the transmit ring is busy, iflib waits for a bigger batch. When it's nearly empty, iflib writes right away so a lone packet isn't left waiting.

The bug is in that last case. Below an eighth full, the required batch size is zero, and the rule was "write if the number waiting is at least the number required". Zero is at least zero, so iflib wrote the doorbell even with nothing waiting, pointing the interface at a spot in the transmit ring it had already reached.

iflib makes that check three times each time it sends: before it starts, after each packet, and at the end. A router that forwards each packet as soon as it arrives sends them one at a time into a nearly empty transmit ring. Every packet paid for three doorbell writes. Two of them were for nothing. A write is a store to a register on the interface, and the core can't move on until the interface has taken it. We timed that: the 4200's Intel I226 interface takes about 2.9 million register writes per second however many cores are writing, so with four cores at it they queue for their turn. At 0.96 Mpps with three writes per packet, the router was asking for exactly that: 2.9 million writes per second into one port.

In the 4200's default setup, one queue per port, one core handles each port's traffic. In our one-way test that one core did all the forwarding. It still paid three doorbell writes per packet, but it ran out of processing time at 0.78 Mpps, before it hit the doorbell limit. There the wasted writes burn CPU time instead of capping throughput.

max = TXQ_MAX_DB_DEFERRED(txq, txq->ift_in_use);

/* force || threshold exceeded || at the edge of the ring */
if (ring || (txq->ift_db_pending >= max) ||
    (TXQ_AVAIL(txq) <= MAX_TX_DESC(ctx))) {
	dbval = txq->ift_npending ? txq->ift_npending : txq->ift_pidx;
	...
	ctx->isc_txd_flush(ctx->ifc_softc, txq->ift_id, dbval);
	txq->ift_db_pending = txq->ift_npending = 0;
	return (true);
}

The fix

The fix (4a0ec469934a) is a single early return: if nothing is waiting, don't write the doorbell register.

Now iflib skips both empty writes and writes the doorbell once per packet. The router forwards 2.57 Mpps in one direction, 2.7 times as fast, and at 0.8 Mpps its cores are 23 % busy instead of 36 %. Packets also wait less once the cores get busy: in a round-trip test on one queue, the original code's typical packet queued for 2 ms from 800 kpps per direction, while the fix still answered in about 130 µs.

One write per batch

Packets leave one at a time, but they don't arrive that way. iflib takes up to 16 packets from the receive ring at once (its default receive budget, the same for every iflib driver) and hands them to the network stack in a single call. With direct dispatch, which the 4200 uses by default, each batch is forwarded before the next is collected.

A second change, now in FreeBSD review as D60416, holds the doorbell while a batch is being forwarded and writes it once when the batch is done. It's controlled by a proposed knob, net.iflib.tx_db_batched, which isn't yet part of FreeBSD. It only holds while the transmit ring is at most an eighth full, the range where iflib's own batching does nothing. Measured on one core, this cuts the count to 0.063 writes per packet, close to one per batch of 16. The charts call this version Batched. The review is also looking at how the change behaves when the machine is a TCP endpoint rather than a router.

Holding the doorbell delays the first packets of a batch. We measured it: a round trip through the router grows by 6 to 16 µs, and that's with a test sender that fills every batch; real traffic at low rates arrives singly and gets its doorbell write at once. Under heavy load the effect reverses: with fewer doorbell writes, the slowest packets get through sooner.

Packets that leave outside a batch, such as those handed to a separate thread or sent by the router itself, still cost one doorbell write each.

With both changes, the 4200 forwards 3.58 Mpps in one direction with its cores 70 % busy, and 5.41 Mpps in both directions at once (bidirectional), up from 1.54.

The results below compare the three versions in the same colours on every chart: Original (three writes per packet), Fix (one per packet) and Batched (one per batch). In the load chart, a line on the diagonal is forwarding everything it's offered and a line that goes flat has hit its limit; the lower panel shows how busy the cores were.

What each write costs

To price a single doorbell write, we had one core do all the forwarding and counted the cycles it spent per packet in three workloads: firewall off, firewall tracking 2,050 states, and firewall tracking about 200,000 states. Each write removed saves 330 to 395 cycles, whatever the firewall is doing.

Removing two writes saves about the same number of cycles whether a packet takes 2,700 cycles to forward or 6,000: a big share of a cheap packet, a smaller share of an expensive one. With the firewall off, the fix speeds up a single core by 42 %; with 200,000 states tracked, by 14 %. Batching roughly doubles those gains.

Where the time goes

A flame graph shows where the processors spent their time. Each bar is a function, its width is its share of all the samples, and the bars stacked on it are the functions it called. The three panels share one scale and were captured on all four cores at 2.0 Mpps offered one way, firewall off.

From one panel to the next, the doorbell bar shrinks and the idle bar grows.

What's left

In both directions, the batched version runs out of CPU at 5.41 Mpps, 73 % of line rate. Part of that limit is heat: above 5.6 Mpps offered, the cores reached 76 °C and throttled, so a cooler box would get further. No single function stands out the way the doorbell did. The largest is memcpy_erms, at 4.0 % of the time: iflib copies each received packet of up to 128 bytes into a fresh buffer so the original can stay in the receive ring. Next is m_free, at 3.6 %, freeing buffers once they have been sent.

In one direction, the router forwards 3.58 Mpps with its cores 70 % busy. The CPU isn't the limit. The 3.7 % that doesn't get through at line rate is dropped while the interface's transmit ring is full: the I226 itself is sending no faster than that. We haven't found what sets that limit.

Nobody runs off anymore

The prank is over. Since 4a0ec469934a, iflib writes the doorbell only when something is waiting. Batched doorbells go further and write it once per batch. Together they took the 4200's one-way forwarding from 0.96 to 3.58 Mpps, and because the fix is in iflib itself, every driver built on it stops pressing the bell for nothing. No function in the profile now costs anything near what the doorbell did, so the next ceiling will have to be found somewhere other than the front porch.