A spinlock bug in eBPF doesn't announce itself with a stack trace or an error message. It announces itself with a machine that stops responding. No logs, no warnings, no graceful degradation — just a CPU core spinning forever on a lock that will never be released. If you're lucky, the watchdog timer fires and you get a kernel panic with a trace. If you're unlucky, the machine just freezes and you have to power-cycle it.
eBPF is supposed to be safe. It runs in the kernel but goes through a verifier that proves programs terminate, don't access invalid memory, and don't corrupt kernel state. So how do spinlock bugs get through? Because the verifier checks individual programs, but spinlock bugs emerge from interactions between programs, maps, and the kernel's scheduling decisions — things no static verifier can fully predict.
What eBPF Spinlocks Are For
eBPF programs run in kernel context, often on multiple CPUs simultaneously. When two eBPF programs need to update the same map value — a counter, a data structure, a state machine — they need synchronization. eBPF provides bpf_spin_lock and bpf_spin_unlock for this.
A spinlock is the simplest possible lock: try to acquire it, and if it's held, spin in a tight loop until it's released. No sleeping, no yielding, no queuing. This is appropriate for eBPF because eBPF programs run in contexts where sleeping is forbidden — interrupt handlers, softirqs, pre-emption-disabled sections. A mutex that puts the caller to sleep would crash the kernel. A spinlock just burns CPU cycles until the lock becomes available.
struct map_value {
struct bpf_spin_lock lock;
__u64 counter;
__u32 last_pid;
};
SEC("tp/sched/sched_switch")
int handle_sched_switch(struct trace_event_raw_sched_switch *ctx) {
__u32 key = 0;
struct map_value *val;
val = bpf_map_lookup_elem(&my_map, &key);
if (!val)
return 0;
bpf_spin_lock(&val->lock);
val->counter++;
val->last_pid = ctx->next_pid;
bpf_spin_unlock(&val->lock);
return 0;
}This looks straightforward. And for simple cases like this — lock, update a value, unlock — it works fine. The problems start when the locking patterns get more complex, or when the interactions between eBPF programs and the kernel's own locking create unexpected dependencies.
How Spinlock Bugs Manifest
The classic spinlock bug is a deadlock: program A holds lock 1 and tries to acquire lock 2, while program B holds lock 2 and tries to acquire lock 1. Both spin forever, waiting for the other. In kernel context, this means those CPU cores are gone — they'll never run anything else.
The eBPF verifier prevents some of these cases. It rejects programs that hold more than one spinlock simultaneously, which eliminates the classic AB-BA deadlock. But it can't prevent all deadlock scenarios because some emerge from the relationship between eBPF spinlocks and the kernel's own locking.
Consider this scenario: an eBPF program attached to a tracepoint acquires a spinlock in a map. While the lock is held, a hardware interrupt fires on the same CPU. The interrupt handler runs another eBPF program that tries to acquire the same spinlock. Deadlock — the interrupt can't return until it gets the lock, but the lock can't be released until the interrupted program resumes, which can't happen until the interrupt returns.
Deadlock scenario (same CPU):
CPU 0 running eBPF program A
→ acquires spinlock on map_value
→ hardware interrupt fires
→ CPU 0 runs interrupt handler
→ interrupt handler runs eBPF program B
→ program B tries to acquire same spinlock
→ spinlock is held by program A
→ program B spins... forever
→ interrupt handler never returns
→ program A never resumes
→ program A never releases lock
→ CPU 0 is permanently stuckThe kernel's own spinlocks handle this by disabling interrupts while a spinlock is held (spin_lock_irqsave). eBPF spinlocks do the same — bpf_spin_lock disables preemption and, on kernels 5.1+, disables softirqs. But not all interrupt contexts are covered, and not all kernel versions handle this identically, which is where the subtle bugs live.
The Debugging Problem
Debugging spinlock issues in eBPF is hard for specific reasons.
Reproduction is non-deterministic. Spinlock bugs depend on timing — which CPU runs which program, when interrupts fire, how long the critical section takes. A test that runs fine 999 times might deadlock on the 1000th. You can't reliably reproduce them in development because your development machine has different CPU topology, interrupt frequency, and load patterns than production.
The failure mode destroys evidence. When a spinlock deadlock occurs, the affected CPUs stop running anything — including the logging and monitoring infrastructure that would tell you what happened. If the deadlock is on a single CPU, the machine might limp along. If multiple CPUs deadlock (or if the locked CPU holds other resources that other CPUs need), the entire machine hangs.
Traditional debugging tools don't help. You can't attach a debugger to the kernel to inspect the deadlocked state (well, you can with KGDB, but you need a serial connection set up before the deadlock). dmesg is useless because nothing is writing to it. The only reliable information comes from the lockup detector's output — if it fires before the machine dies completely.
Strategies That Actually Work
Despite these challenges, people do find and fix eBPF spinlock bugs. Here's how.
Lock ordering analysis. Before running anything, analyze every eBPF program's locking behavior. Which maps does each program access? Which locks does it acquire? Can two programs that touch the same lock run on the same CPU (same tracepoint, or tracepoint vs. interrupt)? This is tedious manual analysis but catches the most common deadlock patterns before they happen.
Lock hold time monitoring. Instrument your eBPF programs to measure how long spinlocks are held. A spinlock held for more than a few microseconds is a red flag — it increases the window for interrupt-based deadlocks and causes other CPUs to waste cycles spinning. Short critical sections are both faster and safer.
// Monitoring lock hold time in eBPF
SEC("tp/sched/sched_switch")
int handle_switch(void *ctx) {
struct map_value *val;
__u64 start, elapsed;
__u32 key = 0;
val = bpf_map_lookup_elem(&my_map, &key);
if (!val)
return 0;
start = bpf_ktime_get_ns();
bpf_spin_lock(&val->lock);
// Critical section — keep this minimal
val->counter++;
bpf_spin_unlock(&val->lock);
elapsed = bpf_ktime_get_ns() - start;
// Report if lock was held too long
if (elapsed > 1000) { // > 1 microsecond
bpf_printk("lock held for %llu ns", elapsed);
}
return 0;
}Per-CPU data structures. The best spinlock bug is one that can't happen. If each CPU works on its own copy of the data and you aggregate later, there's no contention and no locks needed. eBPF's BPF_MAP_TYPE_PERCPU_HASH and BPF_MAP_TYPE_PERCPU_ARRAY are designed for exactly this. Counters, histograms, and accumulators almost never need shared-state spinlocks.
Atomic operations instead of locks. For simple operations — incrementing a counter, comparing and swapping a value — __sync_fetch_and_add and similar atomics are faster and deadlock-free. You don't need a spinlock to increment a counter. Reserve spinlocks for operations that genuinely need multiple fields updated atomically.
What This Reveals About eBPF's Safety Model
eBPF's safety story is impressive but nuanced. The verifier guarantees that individual programs are safe: they terminate, they don't access invalid memory, they don't corrupt kernel state. But 'safe' programs can compose into unsafe systems when their interactions create timing dependencies the verifier can't analyze.
This is a fundamental limitation of static analysis. The verifier sees each program in isolation. It doesn't know which other programs are loaded, which maps they share, or what kernel contexts they run in. A program that acquires a spinlock is 'safe' — it acquires and releases correctly. But whether that program is safe in combination with every other loaded eBPF program depends on runtime conditions.
The kernel community has been addressing this incrementally. Recent patches have added restrictions on which eBPF program types can use spinlocks, how long critical sections can be, and what operations are forbidden while a spinlock is held. Each restriction narrows the window for bugs but also limits what eBPF programs can do.
Practical Rules for eBPF Locking
After debugging enough spinlock issues, patterns emerge for avoiding them.
- Prefer per-CPU maps. If your data can be partitioned by CPU and aggregated in user space, do that. No locks, no contention, no deadlocks. This handles 80% of use cases — counters, event logs, histograms.
- Prefer atomics over spinlocks. If you need a shared counter or a compare-and-swap, use atomic operations. They're lock-free and can't deadlock.
- If you must use spinlocks, keep critical sections tiny. Increment a counter, update a timestamp, swap a pointer. Nothing more. Never call helper functions while holding a lock — you don't know what locking they do internally.
- Never hold the same lock from two eBPF program types. If a kprobe program and a tracepoint program both lock the same map value, you're one interrupt away from a deadlock. Use separate maps or per-CPU data.
- Test under load. Spinlock bugs are timing-dependent. They surface under contention — high CPU utilization, frequent interrupts, many eBPF programs running simultaneously. Test with realistic production load, not idle development machines.
eBPF has fundamentally changed how we interact with the Linux kernel, giving user-space programs safe access to kernel-level observability and networking. But 'safe' has boundaries. The verifier makes eBPF safer than writing raw kernel modules — dramatically so. It doesn't make it safe the way user-space programming is safe. Understanding where those boundaries are, especially around shared state and locking, is what separates eBPF programs that work in production from ones that work in testing.