NextG Systems

A 5G cell on an M.2 card: bringing up the LiteX M2SDR with OCUDU

2026-10-06T00:00:00Z

OCUDU gives the community an open, carrier-grade CU/DU. The radio underneath it has mostly been a USRP. We wanted to know whether a small, fully open radio could carry a real cell: the LiteX M2SDR from Enjoy-Digital, an M.2 2280 card with an Artix-7 XC7A200T and an AD9361, open from the gateware up to the SoapySDR driver.

It can. With the changes described here, an OCUDU gNB runs on the M2SDR on band n78 (TDD, 30 kHz SCS, 20 MHz, 23.04 MSps) with OCUDU's stock lower PHY and default timing, plus one static RX timestamp correction. A commercial phone attaches and holds the link:

14 minsoak with a phone attached and pinging

0.003 %TX frames late (292 of 9.5 million)

0RX discontinuities, “PUxCH request late” or “UL processor is busy”

42samples RX-to-TX offset, identical on 9 of 9 cold starts

Getting there was almost entirely about time: when a sample leaves the antenna, what timestamp a received sample carries, and how late the host may be before something breaks. None of it was exotic, but each piece hid behind the next. This post walks through them in the order we found them. All of the work is on the hw-timed-tx branch of our fork, with the full design notes in doc/timed_tx_hw.md and doc/rx_delivery_lag.md.

What a gNB expects from a radio

We connect OCUDU to the M2SDR through a SoapySDR radio driver we added to OCUDU (upstream ships UHD and ZMQ); it lives in lib/radio/soapy/ on the soapy branch of our fork. Whatever the driver, OCUDU, like srsRAN before it, assumes UHD semantics:

Timed TX. The gNB writes each burst with a timestamp. The radio holds it until that instant, emits it on exactly that sample, and drops it if it arrives late. The timeline never moves.
Exact RX timestamps. Every received buffer is labelled with the index of its first sample, and consecutive labels are contiguous.
Back-pressure. A write blocks once the radio holds enough not-yet-due samples, so the gNB cannot run arbitrarily far ahead.

A UE synchronises to the downlink it actually receives, then schedules its uplink against that. If the radio emits downlink samples a few microseconds away from where the gNB thinks it did, the uplink comes back off the gNB's slot grid. PRACH tolerates this: the format we use aliases the error modulo 33 µs, so the phone still gets through random access. PUSCH does not tolerate it, and the cell looks almost-working: attach starts, uplink decodes fail.

Starting point: timing in software

The M2SDR's original driver timed TX in software. It streamed through a free-running ring of 256 DMA buffers of 2048 samples each and anchored "sample N leaves at board time T" once per start, and again after every underflow. That anchor has one-buffer granularity, a race with the DMA start, and a whole-lap ambiguity (the ring is 22.76 ms at 23.04 MSps). So every start, the real emission time differed from the stamp by a random offset, and every re-anchor moved it again.

It was possible to work around this on the OCUDU side by measuring the offset at start-up and shifting RX timestamps to match. That got a phone attached, but it is a patch over a radio that doesn't keep time. Hardware-timed radios (UHD, LimeSDR, bladeRF in metadata mode) don't need it, so we set out to give the M2SDR the same semantics.

A timed TX gate in the FPGA

Earlier attempts existed. A scheduler inside the TX header module (late 2025) never streamed in the right frame. A sys-domain hold/pass/drop arbiter (spring 2026) worked logically but sat upstream of the AD9361 core's TX FIFO, which has priming hysteresis: it starts output only when half full and stops when empty. Anything released upstream of it lands in a FIFO whose fill time depends on DMA burst timing, so the release jitter was microseconds and every gap re-primed it.

The new design, TimedTXGate, sits between the TX header extractor and the loopback mux. Each 8 KiB DMA buffer already carries a 16-byte header with a 64-bit timestamp. On the first payload word of each frame the gate decides:

condition	action
no timestamp, or gate disabled	pass (untimed)
due in the future	hold: stall the DMA reader; the PHY outputs zeros until the frame is due
late by less than the margin	pass
late by more than the margin	drop the frame and count it

While the gate is enabled, the SoC forces the RFIC TX FIFO into its started state, which removes the priming hysteresis. After a hold the downstream FIFOs are empty, so release-to-air latency is a fixed pipeline depth. In a continuous stream only the first frame after a start is actually held; the rest arrive on time and pass. After a host stall, the frames that missed their time are dropped and the stream re-aligns on the first on-time frame. No re-anchor, no timeline shift.

On the host side, the SoapySDR plugin gets a timed_tx=hardware device argument: it enables the DMA header, stamps every buffer with the board time of its first sample, and reports frames the gate dropped as SOAPY_SDR_TIME_ERROR.

What the air taught us

The gate passed simulation on the first day. Over the air, with OCUDU on top, it took a series of fixes before it was visible at all. Every one of these was in the driver or the plugin, not the gate:

The DMA synchronizer waited for PPS. The kernel armed the TX DMA to start on the next PPS edge, up to a second later. Every frame stamped less than a second ahead was already late. Hardware mode now bypasses the synchronizer; the stamps carry the timing.
The DMA reader prefetches, and the ring goes stale. The reader free-runs over the ring and prefetches two or three buffers ahead of its index. Slots not rewritten since the previous lap get re-read with their old stamps. The fix: zero the ring before starting (a zero header means untimed silence), and keep the write pointer a few slots ahead of the reader's live index rather than the kernel's count, which only refreshes every 8 buffers.
One part per million of rounding. OCUDU stamps only the start of a burst; later writes are stamped from the plugin's own timeline, which advanced by llround(n × 1e9 / rate) nanoseconds per chunk. That 0.3 ns per chunk is about 1 ppm, so emission drifted ~20 samples per second until the burst restarted. The timeline is now anchor plus an exact sample count.
The late margin becomes pipeline fill. A frame that arrives late but within the margin is emitted immediately, and its lateness stays in the TX pipeline as a fill level that never drains, because every later frame is then held until its stamp. With a one-frame margin the offset settled anywhere within a frame (1866 vs 168 samples on two starts). We cut the margin to 1 µs, so every frame goes through the hold path; the fine gate described below later let it grow again.
Stale re-reads are not late frames. OCUDU ends its TX burst on every TIME_ERROR. The resulting gap made the reader sweep stale slots, whose drops were reported as more TIME_ERRORs: a self-sustaining loop after any host stall. Stale drops now go to their own counter and are not reported to the application.
Loopback can't measure what matters. In the AD9361's PHY loopback, RX only produces samples while TX flows, so RX time freezes during a hold. Loopback proves hold, release, continuity and drops; the release offset has to be measured over the air.

The bug was on the receive side

With TX fixed, the cell still had a start-dependent failure: on about half of the starts the phone attached and then the uplink died, with OCUDU logging a permanent storm of "PUxCH request late" and "UL processor is busy". It looked like a TX timing problem. It wasn't.

RX frames were labelled with each frame's FPGA nanosecond timestamp. Those stamps carry quantisation noise, so converted back to samples the labels were not always contiguous: off by ±1 sample, 2,000 to 3,000 times per second, depending on the sub-sample phase at start. OCUDU's lower PHY treats any such mismatch as lost alignment and discards RX up to the next subframe. Across 8 starts the correlation was exact: zero label jumps meant zero late requests; 42k to 66k jumps meant the storm.

The fix was to stop using nanoseconds as the source of truth.

Labels by counting. The plugin labels RX buffers by counting samples from an anchor, using the FPGA stamp only to detect real gaps.

A sample counter in the gateware. Then we moved the counting into the FPGA. A 64-bit counter in the RFIC clock domain advances on every PHY word by the number of samples it carries. Each RX word travels through the clock-domain FIFO and pipeline together with its tick, and the header inserter stamps each frame with the tick of its first sample. On TX, the sys-domain gate still decides hold, pass or drop per frame, but now passes the frame's stamp through a small clock-domain FIFO to a new TXFineGate right in front of the PHY. That gives every word its own tick and emits it in the PHY slot carrying that tick. A word whose slot has passed is discarded; an early word waits. The stream can lose samples but is never shifted.

The TX path on hw-timed-tx. The coarse gate decides per frame; the fine gate places every word on its exact sample tick in front of the PHY.

Two more details closed the loop. The PHY's TX word counter now follows the RX word counter instead of free-running, so the RX-sample-to-TX-slot relationship is the same after every initialisation. And with time_base=samples, every timestamp crossing the SoapySDR API is the sample tick itself rather than nanoseconds, so nothing depends on rounding.

Timed RX start and back-pressure. OCUDU seeds its timeline from "RX begins at init_time", but the plugin started RX about 95 ms earlier. A small gateware block now holds the RX header inserter until the requested tick, so the first frame delivered is that sample. And where a USRP blocks send() once its FIFO is full, the M2SDR accepted 22.7 ms of samples; OCUDU stamps its first frames 100 ms ahead, so its downlink ran 177 to 194 slots ahead of RX at start. The plugin now bounds the number of submitted-but-not-due frames (default 96, about 8.5 ms) and writeStream times out beyond that.

Result: on every start, the first RX frame is exactly OCUDU's init_time; zero non-contiguous labels and zero late uplink requests on 8 of 8 starts; the RX-minus-TX offset measured on the SSB is 42 samples on 9 of 9 cold starts, each including an AD9361 re-initialisation. A constant offset is just a calibration value: OCUDU applies it once as a static RX timestamp shift, and the residual offset measures 0 samples.

Latency: from 472 µs to 8 µs

With timing deterministic, the next limit was the latency budget. OCUDU's lower PHY releases a downlink slot when the newest RX timestamp reaches the slot start minus rx_to_tx_delay (1 ms by default). Every microsecond the RX path spends delivering a buffer comes out of the time TX has to reach the FPGA.

Measured against the gateware sample counter, the age of the newest sample in each buffer when the receiving thread got it had a median of 472 µs. Two causes:

One interrupt per 8 buffers. Buffers arrived in batches of 8, the first of each batch already 621 µs old.
The completed-buffer count was one short. The DMA table's status is the index of the descriptor that completed last, not a count. The driver used the bare index, so the newest complete buffer was withheld until the next interrupt.

Fixing both (one interrupt per RX buffer by default, count = index + 1, and POLLIN as soon as one buffer is ready) brought the median to 8 µs:

mode	p50	p99	p99.9	IRQ/s
before: 8 buffers/IRQ, count one short	472 µs	724 µs	764 µs	2.8 k
1 buffer/IRQ, count fixed (new default)	8 µs	38–46 µs	204–320 µs	12.7 k
busy-poll (`rx_poll=busy`)	4 µs	8–22 µs	130–208 µs	1.5 k

As seen by OCUDU's lower PHY, one thread hop later, that's 355 µs average before and 20 µs now.

Do we need a DPDK-style poll-mode driver?

For RX delivery, no. Busy-polling halves the median (8 → 4 µs, against an 88.7 µs frame and a 1 ms budget) and costs a full core. The p99.9 and the maximum don't move at all: on our test laptop they are set by firmware stalls of up to ~0.2 ms at least once a second, measured with the hwlat tracer. A poll-mode driver only improves the tail on a host prepared for it, with isolated cores and quiet firmware. The stream itself is 11,272 frames/s, nowhere near the packet rates DPDK exists for.

The last piece: a TX DMA that stops

With RX fast, OCUDU was handing each downlink slot to the driver about 940 µs ahead of its time. That should be plenty. It wasn't, because of what happened when the host was late once.

The TX DMA ran in LitePCIe's loop mode: the reader free-runs around a 256-slot ring, paced only by the gate holding the frame at its head. When a host stall let the reader catch up with the write pointer, nothing held it any more. Every slot ahead was a frame from the previous lap, so the gate dropped them as stale at PCIe speed; the reader lapped the ring every ~2 ms, and the host, writing 11 frames per millisecond, could never get ahead of a reader moving 125 slots per millisecond. One stall of a millisecond turned into an 80 ms episode costing ~500 late and ~4000 stale frames. With the default 1 ms delay, episodes were continuous; we had been running OCUDU with a 4–5 ms delay to stay clear of them.

What TX needed was FIFO semantics: the reader should stop when there's nothing new, like a USRP's flow-controlled TX FIFO. LitePCIe already supports this through its descriptor table's prog mode, which executes each descriptor once and stops when the table is empty. No gateware change was needed: a new ioctl selects FIFO mode, and each submit queues one descriptor per buffer. A frame is now fetched as soon as it's submitted, nothing is ever read twice, and a frame submitted too late is simply dropped by the gate.

`rx_to_tx_delay`	loop mode	FIFO mode
5 ms	clean	clean
2 ms	episode when the phone attached	0 late, 0 stale
1.5 ms	8–12 episodes per 45 s	0 late, 0 stale
1 ms (OCUDU default)	continuous episodes, unusable	0 late, 0 stale

That is the configuration behind the numbers at the top: stock OCUDU timing, 14 minutes with a phone attached and pinging, 292 late frames out of 9.5 million, ping 2769/2800. Each host stall now costs exactly the frames that were late and nothing more. HARQ error rates dropped too, from DL 3–19 % and UL 10–28 % with the 4 ms workaround to DL ~4 % and UL 4–9 %.

Along the way, every OCUDU-side workaround came back out. Earlier in the bring-up we had a configurable RX-to-TX delay in the lower PHY, a larger UL request pool, a wider RA response window and a raised max_proc_delay. All of them are gone. OCUDU's lib/phy is identical to upstream; the radio driver is the only OCUDU change the M2SDR needs, which is how it should be.

Try it

Two branches, both public:

Radio: hw-timed-tx in our M2SDR fork — gateware, kernel module, libm2sdr and the SoapySDR plugin.
gNB: soapy in our OCUDU fork — the SoapySDR radio driver (lib/radio/soapy/), the cell config and the operational scripts. The full bring-up notes are in docs/m2sdr_soapy_bringup.md.

Build and load the radio side:

# Gateware (Vivado 2026.1)
python3 litex_m2sdr.py --variant m2 --with-pcie --pcie-gen 2 --pcie-lanes 4 --build

# Flash, reload, then rescan the PCIe bus
m2sdr_util flash-write <bitstream>_operational.bin
m2sdr_util flash-reload

# Rebuild and reload the kernel module (litex_m2sdr/software/kernel),
# rebuild user/ and the SoapySDR module, install them, and check:
SoapySDRUtil --find

Build OCUDU from the soapy branch as usual, then start from configs/gnb_soapy_m2sdr_hwtimed.yml. The radio section is the interesting part:

ru_sdr:
  device_driver: soapy
  device_args: driver=LiteXM2SDR,ad9361_fir_profile=bypass,rx_worker_packets=2048,freq_corr_ppm=1.327,timed_tx=hardware,time_base=samples
  srate: 23.04
  tx_gain: -10      # the driver maps TX gain to AD9361 attenuation: 10 dB here
  rx_gain: 40
  expert_cfg:
    tx_mode: continuous

freq_corr_ppm is the crystal error of our board; measure yours. timed_tx=hardware also selects the FIFO TX queue and the gateware sample counter.

The one runtime setting is the static RX timestamp correction, passed as an environment variable:

OCUDU_SOAPY_RX_TS_SHIFT=-42 ./apps/gnb/gnb -c configs/gnb_soapy_m2sdr_hwtimed.yml

42 samples is the over-the-air value for our board at 23.04 MSps; scripts/m2sdr/align.py measures it from TX and RX captures (keep the phone in airplane mode while you measure, or its uplink swamps the SSB the tool correlates on). The phone registers even without the shift, since 2 µs is well inside the PRACH detection window. For long runs, scripts/m2sdr/guardian_hw.sh starts the gNB with the shift and restarts it on real-time trouble.

While the cell runs, the gate's counters should show late and stale flat and passed tracking the frame rate (11,272 frames/s at 23.04 MSps). Without RF, software/soapysdr/test_hw_timed_tx.py --lead-ms 300 --reps 4 --late-test in the M2SDR repo exercises hold, release, sample continuity and late drops through the AD9361 PHY loopback.

What's next

Upstream. This all started from Enjoy-Digital's excellent base design, and the goal is to get the timed TX gate, the sample counter and the driver fixes back into enjoy-digital/litex_m2sdr, and the SoapySDR radio driver into OCUDU.
Higher rates. At 61.44 MSps 2R2T a DMA frame is 16.6 µs, which would mean 60k interrupts per second at one interrupt per buffer. A larger interrupt period or busy-polling is the answer there.
Better hosts. Everything above was measured on a six-core laptop with SMI-heavy firmware and Wi-Fi that re-associates every half hour. The remaining tail is the host, not the radio.
Lower user-plane latency. With deterministic timing, the margins can shrink. On a PREEMPT_RT kernel with max_proc_delay at its minimum of 1, ping RTT through the cell went from 28 ms with the default configuration to 12 ms (one-way UL 21.8 → 7.75 ms, DL 4.6 ms), with no loss. Experiments with an event-driven downlink go further and are on the low-latency-phy branch; that's a post of its own.

We'll be at the OCUDU developer summit and happy to talk through any of this, or to see an M2SDR running your cell.

Stop asking, start predicting: SAGE cuts 5G uplink latency by 2.5×

2026-10-07T00:00:00Z

Cellular networks got fast a while ago. What they never got is prompt, and most of the delay is on the way up: before a phone may send a single uplink byte, it has to ask the base station for permission, and usually it has to ask twice. Our paper SAGE: A Real-Time AI System for Reducing Latency in NextG Cellular Networks, written with Haitham Hassanieh's group at EPFL and published at ACM SIGCOMM 2026 in Denver, takes that waiting out of the loop. The gNB learns each user's traffic, predicts when the next uplink burst will arrive and how big it will be, and has a grant ready when it does.

It needs no change to the 3GPP protocol, the phone or the radio: SAGE is software at the gNB and its RAN controllers. On a real over-the-air testbed with commercial 5G modules and eight real applications:

2.53×lower uplink latency than grant-based access, on average

1.59×the radio resources of grant-based access, on average

37.5×fewer resources than grant-free baselines (up to), at comparable latency

51 µsaverage inference time per prediction, on one CPU thread

This post is the short version: why the uplink is slow, why predicting it is harder than it sounds, and what we built. The paper has the full design, the analytical model and many more experiments.

Why the uplink waits

A 5G phone with new data can't just transmit. In grant-based access it waits for its next Scheduling Request (SR) opportunity, which can be as far as 40 ms away in commercial networks, and sends an SR. The gNB answers with a small fixed grant, just enough for a sliver of data and a Buffer Status Report (BSR) saying how much is still queued. Only then does the gNB grant the rest. That is at least two request/grant round trips before the bulk of a burst moves.

Grant-based access spends two request/grant round trips before most of a burst moves. If the gNB knows when the burst is coming and how big it is, it can schedule the grant in advance and skip both.

The alternative, grant-free access, pre-allocates uplink resources to the phone in every slot. Latency drops, but the resources sit idle whenever there is nothing to send, and the approach does not scale with many users. So operators choose between keeping devices waiting and wasting spectrum.

Predicting traffic breaks that trade-off: allocate just the right amount, just in time. Prior work on cellular traffic prediction mostly looked at minutes to days, for planning and load balancing. The few real-time systems we know of handle periodic traffic with payloads of a few hundred bytes or less, or need a GPU and tens of milliseconds per inference. We wanted bursty, non-periodic, real-application traffic: video, AR, games, video calls, industrial control. Predicted at millisecond granularity, on a CPU.

The gNB doesn't see the traffic

The first obstacle is that the gNB never observes what actually arrives at the phone. It sees SRs, BSRs and decoded payloads (SDUs), all shaped by the protocol configuration, the scheduler and the channel:

One burst becomes many events. A burst that doesn't fit in one slot's grant is fragmented into several SDUs and triggers several BSRs. Change only the channel bandwidth, 40 to 60 MHz, and the same 35 KB burst produces a completely different sequence.
BSRs are quantized. A BSR reports a buffer level, not a byte count: anything from 3910 to 5446 bytes reports as index 20. Subtract consecutive reports and you invent "ghost arrivals" of data that never existed.
Retransmissions move things in time. A transport block that fails decoding shows up at the gNB later than it was sent.

Train a predictor on this raw view and it learns the network's quirks rather than the application.

Traffic trains

Our answer is an abstraction we call a traffic train. The gNB first derives arrivals from pairs of consecutive BSRs and the SDUs received in between, then groups closely spaced arrivals into one train, ended by a configurable idle gap (the train detection threshold, a few UL slots). Short thresholds suit URLLC slices whose bursts fit in a slot; longer ones aggregate eMBB bursts that span several.

A few details make trains robust:

Size from the lowest BSR. A train's size is the SDUs delivered up to its final BSR, plus that BSR, minus the last BSR before the train. The final BSR is the smallest level seen, so quantization error is minimized. And since size estimation is decoupled from detection, a train can be closed early, for timely prediction, without waiting for the rest of its SDUs.
SRs open trains early. An SR after an idle period starts a provisional train, timed at the SR, before any BSR arrives.
HARQ as ground truth for time. HARQ remembers the slot of each transmission's first attempt. When a retransmitted SDU finally decodes, SAGE puts it back where it belongs: it can create a train that was missed, correct an existing train's time or size, or merge two trains a lost packet had split.

The result is a sequence of burst-level events that tracks what the application sent, not how the RAN happened to carry it. That matters for accuracy (in a micro-benchmark on mobile AR and SLAM, predictors trained on trains were much more accurate than ones trained on raw derived arrivals) and for portability: with two national operators' configurations we measured 3.2–4.1× lower latency than grant-based access without retraining any model.

One small model per traffic profile

The second obstacle is diversity. A single universal predictor big enough for every application is too slow for millisecond deadlines and expensive to retrain each time a new application shows up, and retraining it for a new profile can degrade the old ones.

SAGE instead keeps a traffic-aware model database of small, profile-specific predictors. Each profile is keyed by a 16-number fingerprint, eight statistics (mean, median, IQR, 10th and 90th percentile, skewness, MAD, lag-1 autocorrelation) of inter-arrival times and eight of sizes. A new user's trains are fingerprinted and matched to the nearest profile; the lookup scales to 10,000 profiles in 200 ms on an Intel NUC. The predictors in our evaluation are LSTMs of 3,490 to 50,818 parameters. Predictors are plug-ins, so a profile can use anything from EWMA to a transformer; in our comparison, modern learned predictors performed similarly to each other and consistently beat EWMA.

The control loop

SAGE spans the gNB, a real-time RIC co-located with it, a near-real-time RIC and the model database. A new user goes through five stages:

Traffic characterization. The user starts in ordinary grant-based or grant-free mode while the gNB records its trains. The near-RT RIC tests the series for stationarity; if it is predictable, it retrieves the matching model.
Model verification. The model predicts in the shadows and the gNB scores it against the trains that actually arrive.
Prediction and scheduling. Once verified, the user switches to prediction mode. The gNB pre-allocates resources over a window around the predicted slot, asymmetric because bursts start at the predicted slot and spill forward: fewer resources before it, more after. The window widens when predictions start missing and shrinks when they are good, with two thresholds for hysteresis.
Continual learning. If verification fails, the model is fine-tuned online, one epoch per new train, and re-verified.
Database update. If that still fails, the near-RT RIC trains a new model on a longer series and registers it as a new profile.

Traffic that stays unpredictable falls back to grant-based or grant-free access, and for grant-free an analytical latency/resource model picks the guaranteed grant size at the knee of the trade-off curve. Prediction is used when it's reliable, suspended when it isn't, and improved over time.

The real-time parts are cheap. On a single thread of an AMD Ryzen 7 7840HS, inference averages 0.051 ms and a continual-learning step 0.451 ms. Against an average inter-arrival time of 59.4 ms across our applications, one thread fits over a thousand inferences between consecutive trains.

Results

We built SAGE on srsRAN, with Open5GS as the core and our own RICs, all containerized. The gNB and RT RIC run on a Dell Precision 5860 with a USRP X410 radio, band n78, and the UEs are commercial 5G modules. Every experiment is over the air. The applications: surveillance video, mobile AR, SLAM on a mobile robot, Counter-Strike 2, Google Meet, WhatsApp video calls, an industrial-control trace and a SCADA (Modbus/TCP) trace.

Scenario	Uplink latency vs. grant-based	Resources vs. grant-based
Eight applications, one UE each (average)	2.53× lower	1.59×
Ten concurrent commercial UEs, four applications	1.42–2.41× lower	1.15–1.55×
Two national operators' configurations	3.2–4.1× lower	1.4–1.5×

A few findings worth calling out:

Averages don't work for bursty traffic. Granting a 2 Mbps application its average rate is about 500 bytes per slot, nowhere near a 40 KB burst. Granting 4× the average still loses on latency and wastes resources while idle. SAGE matches the grant-free baselines on latency with up to 37.5× fewer resources.
Bigger grants can be slower. For small-payload traffic (Counter-Strike 2, industrial control, SCADA), oversized grants get filled with MAC padding that both ends must process, so even "full grid" grant-free loses to a right-sized grant. Below about 0.1 Mbps, SAGE can even use fewer resources than grant-based access, whose fixed SR grant (512 bytes in srsRAN) over-allocates for tiny packets.
It holds up with many users. With ten UEs from four vendors, Jain's fairness index stayed above 0.99 under both proportional-fair and round-robin schedulers; SAGE submits its grants through the operator's existing MAC scheduler rather than replacing it. Fixed equal-share grant-free sometimes did worse than grant-based, because a burst larger than its reservation queues even while other UEs' reservations sit idle.
It holds up on bad channels. Up to 20% BLER, SAGE stayed close to full-grid grant-free on latency while using resources close to grant-based. Saturating the uplink with best-effort traffic in another slice left SAGE's latency essentially unchanged.
Timing matters more than size. Median arrival-time error was under 4 ms, median size error under 20%. The prediction window absorbs size errors; a missed slot is what costs latency.
Applications feel it. For streamed video, per-frame network latency fell 2.07–2.54×, and end-to-end display latency 1.13–1.23×.

The cost is honest: in steady state, SAGE over-provisions by a constant factor (the ~1.6× above), which reduces how many users a cell carries before airtime saturates compared with pure grant-based access. The paper's appendix models that overhead as a function of user count and bitrate, so an operator can decide per slice where it's worth paying.

Where this fits

SAGE continues a line of work with the same group on what latency open 5G can actually deliver: Ultra-Reliable Low-Latency in 5G: A Close Reality or a Distant Goal? (HotNets '24), Towards URLLC with Open-Source 5G Software (OpenRIT6G '25), and LatencyScope, a system-level model of 5G RAN latency. The recurring lesson is that the slow parts are protocol and scheduling, not the radio's raw speed.

That's also why it pairs with our M2SDR and OCUDU work. There, the job was to make an open radio keep exact time, so the gNB's timeline can be trusted. SAGE works one layer up: once the timeline is trustworthy, the scheduler can stop waiting for the phone to ask.

Open directions from the paper: better train detection under long SR periods, adding channel prediction to size grants and windows, and coordinating predictions across slices.

Read more: the paper (open access, CC BY 4.0) and the project page at EPFL.

Authors: Aoyu Gong and Raphael Cannatà (co-first authors), Arman Maghsoudnia, Néstor Lomba Lomba and Haitham Hassanieh (EPFL), and Dan Mihai Dumitriu (Pavonis).

Why Now: Open Source, AI, and Open Hardware Meet NextG Wireless

2026-10-07T00:00:00Z

This is the first post in a series about what we're building at NextG Systems and why. Future posts will go deeper on the technical pieces.

Who we are

NextG Systems LLC builds open, programmable, and secure NextG wireless systems. Our focus is mission-critical communications — defense, emergency services, and other settings where a network has to be trustworthy, not just fast.

We come to wireless from an unusual direction. Our team spent more than a decade working on software-defined networking (SDN) in the data center and the cloud. Over the last several years we've turned that experience toward 5G: standing up private 5G testbeds with commercial radio units, running the major open-source cores (Open5GS, free5GC, SD-Core), building RAN intelligent controller applications, and working deeply with the srsRAN codebase and its successor, OCUDU. We also work closely with academic collaborators, including the wireless research group at Princeton University.

We've seen this movie before

When we started in data center networking, the network was a set of sealed boxes. Switches and routers came from a handful of large vendors, bundled hardware with software, and exposed whatever management interfaces the vendor chose to support. If you wanted a new feature, you filed a request and waited for the next release.

Over roughly a decade, that changed completely:

The control plane was separated from the data plane, so the logic that decides where traffic goes could run as ordinary software on ordinary servers.
Open protocols replaced proprietary ones at the boundaries between components.
Merchant silicon made high-performance switching ASICs available to anyone, so hardware stopped being a moat.
Open network operating systems let operators run their own software on the switch itself.
Model-driven configuration (OpenConfig, gNMI, and friends) replaced screen-scraping CLIs with structured, programmable interfaces.

The result was not just cheaper networks. It was networks that operators could reason about, automate, test, and change on their own schedule. The large cloud providers built their businesses on this.

The same pattern is arriving in wireless

More than four years ago, the emergence of O-RAN made us think wireless was next. O-RAN defined open interfaces between pieces of the radio access network that had always been sold as one integrated product. That is the same move as separating the control and data planes: once the boundaries are open, each piece can be built, replaced, and improved independently.

Open-source software already existed. OpenAirInterface and srsRAN offered full RAN stacks, and several open-source 5G cores were maturing. But software alone doesn't make an ecosystem, and for a few years momentum built slowly. Running these stacks was a research exercise, not a deployment option.

We started building on srsRAN anyway, because we believed the direction was right. That bet has paid off.

OCUDU changes the game

In 2025 the U.S. Department of Defense's FutureG office selected SRS and DeepSig to lead OCUDU — Open RAN Centralized Unit and Distributed Unit — an open-source 5G/6G RAN initiative built on the srsRAN lineage. OCUDU now lives at the Linux Foundation, with an ecosystem foundation whose members include operators, chipmakers, and equipment vendors. In May 2026, Qualcomm joined as a Premier Member, alongside companies like AMD, AT&T, Ericsson, Nokia, NVIDIA, Samsung, and Verizon.

This is what was missing. Sustained government backing provides a long-term anchor customer and a reason for the community to invest. Vendor participation means the open stack will run on real hardware and be tested against real requirements. For the first time, an open-source RAN looks less like a lab tool and more like the foundation of a product category.

Merchant silicon, with a RISC-V accent

The data center story needed merchant silicon, and so does wireless. That is starting to happen, in fits and starts. Companies like Picocom, EdgeQ, and LeapFrog Semiconductor are building baseband and signal-processing silicon for 5G infrastructure that isn't locked to a single equipment vendor.

What stands out to us is that these designs share a common thread: RISC-V. An open instruction set at the heart of the baseband means the openness can extend all the way down — not just to the software interfaces, but to the processor architecture itself. Combined with open FPGA and SDR platforms, it is becoming possible to build and inspect the whole stack.

Thinking about the whole system

Open components are necessary but not sufficient. Data center SDN succeeded because operators could manage the network as one coherent system. The Open Networking Foundation's Aether project (now hosted at the Linux Foundation) was an early and important glimpse of what that looks like for 5G: an integrated management environment spanning the RAN, the core, and edge compute, with multi-tenancy built in. It showed us what a private 5G network could feel like if it were designed by people who had built cloud infrastructure.

Why AI makes this urgent

The final ingredient is AI, and it cuts both ways.

On one side, AI makes open, programmable networks dramatically more valuable. AI-native signal processing, intelligent RAN control, and automated operations all depend on being able to get inside the stack — something closed systems don't allow.

On the other side, AI raises the stakes on security. As agentic AI makes attacks faster and cheaper, some defenses will have to be just as responsive. But for critical infrastructure, the stronger answer is to eliminate whole classes of vulnerabilities before they can be exploited. That requires systems you can inspect and, increasingly, verify. Open source and open hardware make that possible; closed boxes don't.

There's still a long way to go

We want to be honest: NextG wireless is not where data center SDN is today. Interoperability is still hard. Operations tooling is immature. Performance and reliability of open stacks still lag the best integrated systems in many settings. Silicon options are few. Security assurance for these systems is largely unaddressed.

That gap is the opportunity. We've watched one industry make this journey, and we think we know some of the shortcuts and some of the potholes. NextG Systems exists to help wireless make the same journey, with security and mission-critical reliability built in from the start.

In the coming posts we'll go into specifics: what we've learned running open-source 5G testbeds, where we think formal methods fit in the RAN and core, and what a truly programmable private 5G network should look like.

We're on that journey. Come along.

Slicing an NVIDIA GPU between tenants who don't trust each other

2026-10-14T00:00:00Z

A GPU server shared by several teams is only worth sharing if each team gets the slice it was promised, and none of them can take more by misbehaving. That second half is what existing tools mostly don't do. The common answers either need datacenter hardware with fixed partition sizes, give up isolation, or put the thing doing the limiting inside the container it is supposed to limit.

This post describes how we slice NVIDIA GPUs between mutually untrusting containers. Memory quotas are enforced in gVisor's userspace kernel, the Sentry. Compute shares are enforced by the GPU's own scheduler, driven from the host through a small broker we added to NVIDIA's open kernel modules. The policy is a credit scheduler that a tenant can't game by forking. It runs on consumer and datacenter parts alike: an RTX 5070, an RTX A6000, an A100 and an 8× B300 box. The code is on the gpuslicing branches of our forks of gVisor and of open-gpu-kernel-modules.

0ioctls during 13,000 CUDA kernel launches, which is why compute is hard

0.78 → 2.75share ratio for a 3 : 1 request once the low-weight tenant forks, before and after credit scheduling

99.9 %of solo throughput for a lone tenant with the scheduler running

696 GB/sNCCL all-reduce bus bandwidth across 8 GPUs, inside a gVisor sandbox

Nothing inside the container

The design follows from one rule: no limit may depend on anything inside the container. On a shared GPU the workload is the adversary, and a limit it can reach is a limit it can lift.

HAMi shows why this matters. Its scheduler is genuinely good: it understands requests like nvidia.com/gpumem and nvidia.com/gpucores and places pods so they fit on a device. Its enforcement is libvgpu.so, preloaded into the container to intercept CUDA calls, and it reads an environment variable, CUDA_DISABLE_CONTROL, that switches it off. On a live cluster, the same pod three ways:

HAMi limiter	Sentry limit	Device size the pod sees	Outcome
loaded	none	512 / 512 MiB	refused at 512 MiB
not loaded	none	11347 / 11790 MiB	allocated all 768 MiB it asked for
not loaded	512 MiB	350 / 512 MiB	refused at 320 MiB

Row two is what a tenant gets by setting one variable. Row three is the same pod with the limit enforced somewhere it can't reach. So we kept HAMi's placement unchanged and rebuilt the enforcement elsewhere.

Three pieces, and who trusts whom

gVisor runs a container against the Sentry, a kernel written in Go that runs in userspace, instead of the host kernel. Every syscall the application makes, including every ioctl on /dev/nvidiactl, /dev/nvidia0 and /dev/nvidia-uvm, lands in the Sentry first. Upstream gVisor already proxies these (nvproxy), checking each call against an allowlist of driver structures before forwarding it. That boundary is about the shape of each call. We added quantity.

There are three pieces:

nvproxy in the Sentry enforces memory quotas, because every allocation passes through it.
runsc gpu-scheduler, one process per node on the host, holds the compute policy: who has which weight, and on which GPU.
A broker in the open kernel modules applies that policy to the GPU's own scheduler. Only kernel code can do that.

The split is deliberate. The Sentry is outside the container, but it runs that tenant's code paths, so we treat it as no more trusted than the tenant it serves. Anything that can affect other tenants, like detaching their work from the GPU, lives in the host scheduler and the driver, never in a Sentry.

Memory goes through the Sentry. Compute submission doesn't, so compute is divided below it, by the GPU's own runlist scheduler, under the control of a host process the tenant can't reach.

Memory: quotas in the Sentry

GPU memory is allocated through ioctls, so the Sentry can count it. The idea is simple: charge each allocation against the sandbox's quota and refuse the one that would cross it. The details are where the work went.

Charge before forwarding. The charge is taken before the ioctl reaches the driver and returned if the driver refuses. Otherwise two concurrent allocations could both be admitted against the same headroom. A red-team run with four processes racing allocations behind a barrier summed correctly, with no overshoot.
Count aliases once. NV_ESC_RM_DUP_OBJECT creates a second handle to existing memory, so charges are reference counted. They are released from the cascade that frees an object's dependents, the only place that sees all of them.
Unified memory is charged at reservation. CUDA unified memory is populated by GPU page faults that the host's nvidia-uvm module services internally, without any ioctl the Sentry could see. What the Sentry can see is the address-space reservation, so that is what it charges, together with device memory. It's a sound bound, but a workload that deliberately oversubscribes pays for what it reserved.
Every allocation class makes an explicit decision. A class added to the allowlist without an accounting rule fails a test instead of silently going uncharged.

The Sentry also makes the limit visible. It rewrites the sizes the driver reports, so cuMemGetInfo(), NVML and nvidia-smi inside the sandbox show the quota, not the card. Frameworks size their memory pools from those calls, so an unmodified PyTorch or vLLM fits itself into its slice instead of confidently allocating past it. An over-quota allocation fails with CUDA_ERROR_OUT_OF_MEMORY, exactly as on a full GPU.

Multi-GPU pods needed one more step. HAMi's gpumem is per device, so a pod with two GPUs at 50000 MiB each should get 50000 MiB on each, not 100000 MiB to spend anywhere. Spending it anywhere would let a pod stack its whole allowance on one GPU, into memory promised to another tenant. So nvproxy attributes every allocation to the device it was made on and enforces a per-device limit as well as the total. On a B300, such a pod sees 50000 MiB on each GPU, fits 45000 MiB on both, and is refused 10000 MiB more on either.

In red-team runs, every buffer a tenant allocated next to a victim holding a secret came back zeroed, and out-of-bounds accesses faulted at the GPU's per-context MMU.

Compute: why the Sentry can't do it

The natural next step is to meter compute at the same boundary. It can't be done. Once a CUDA channel is set up, submitting work doesn't enter the kernel: the application writes commands into a ring buffer in memory it has already mapped, then writes to a mapped device register, the doorbell. We measured 13,000 kernel launches producing zero ioctls. There's nothing for the Sentry to intercept.

Our first attempt worked around that, following Krypton (USENIX ATC '25). Submission needs the ring buffer to be mapped, so the Sentry revokes its own mapping of the command buffer when a sandbox's share of each period runs out. The next write faults into the Sentry, which holds the task until its share comes round. Against a kernel-launch loop it divided cleanly: 76.2 %, 51.0 % and 25.8 % of the GPU against 75, 50 and 25 % configured.

Then we ran cuBLAS. On Volta and later, cuBLAS GEMMs and especially CUDA graph replay, which is how an inference server like vLLM spends most of its time, don't rewrite the command buffer per launch. They replay commands written once and ring the doorbell. The revoked mapping is never touched again, so the fault never arrives. Two cuBLAS tenants weighted 3 : 1 under this gate ran at 706 and 706 matmul/s.

The benchmark shared the mechanism's blind spot

The kernel-launch loop that validated the gate is the one workload shape the gate can bind. The clean 76 / 51 / 26 table measured the exception, not the rule. The fix wasn't cleverness, it was testing with the workloads that would actually run.

The gate also had a cost: while it holds a sandbox, the faulting task holds the sandbox's address-space lock, so other threads stall on anything that needs it. A thread doing continuous mmap and munmap fell to about 20 % of its normal rate under a 25 % cap. The gate is still in the tree, but on a node where the broker is running it stands down (more on why below).

The GPU already has a scheduler

The GPU divides its own time. Work is organised in channel groups (TSGs), and the GSP firmware runs a runlist scheduler that rotates between them, giving each a timeslice. It acts below the submission path, so a doorbell can't dodge it. The controls that configure it, setting a channel group's timeslice and removing it from or restoring it to the runlist, are resource-manager calls that need kernel privilege. A container can't issue them. A trusted host component can.

We tried them at kernel privilege with the open driver, and for several days every one appeared to fail. The timeslice control returned NV_OK, and a 16 : 1 setting produced 1 : 1. Detaching a running tenant's channels did nothing. The spatial partition controls returned 0x57, which we read as "not supported." We wrote a careful, detailed conclusion: consumer Blackwell honours none of these, and compute isolation for arbitrary CUDA needs MIG-capable datacenter silicon.

It was three mistakes of our own:

A misread status code. 0x56 is NV_ERR_NOT_SUPPORTED. 0x57 is NV_ERR_OBJECT_NOT_FOUND. The firmware hadn't refused the control; it couldn't find the object the control named.
The wrong call site. We issued the controls from the constructor of the object they named, before it was registered with the firmware. From the deferred scheduling path, after registration, the identical control returns NV_OK.
A missing commit. GSP doesn't act on a staged runlist change until the runlist is restarted. NVA06F_CTRL_CMD_RESTART_RUNLIST is that commit. Without it, both the timeslice change and the detach looked inert.

Each mistake alone produced a plausible negative that reproduced every time. Together, they turned three software bugs into a confident statement about hardware. With the restart added, the levers work, on the A100 first: detaching one of two running tenants handed the survivor the whole GPU (710 → 1551 matmul/s), reattaching restored the even split, and a 16 : 1 timeslice ratio shifted throughput 14 : 1.

The broker

The broker is a small set of hooks in the open kernel modules (in the channel-group and context-share code) that expose one root-only file:

# echo poll > /proc/driver/nvidia/gpusched     # refresh each tenant's GP_PUT
# cat /proc/driver/nvidia/gpusched             # (example output)
pid 41822 active 1 tsgs 3
dev 0000:07:00.0 pid 41822 active 1 tsgs 3
pid 41907 active 1 tsgs 12
dev 0000:07:00.0 pid 41907 active 1 tsgs 12
stats cmds_ok 1840 cmds_failed 0 restarts_ok 1840 restarts_teardown 2 restarts_failed 0 table_full 0

# echo "ts 41822 4000 0000:07:00.0"     > /proc/driver/nvidia/gpusched
# echo "detach 41907 0000:07:00.0"      > /proc/driver/nvidia/gpusched
# echo "attach 41907 0000:07:00.0"      > /proc/driver/nvidia/gpusched

poll makes the driver re-read each channel's GPFIFO put pointer, and reading the file then reports, per tenant and per GPU, whether any of its channel groups advanced (a doorbell-aware activity signal the tenant can't forge) and how many channel groups it holds. ts sets a tenant's timeslice, and detach and attach remove its channel groups from the runlist or restore them. The driver commits each with RESTART_RUNLIST, from an RM work item that holds the GPU locks. The optional PCI address restricts a command to one GPU.

Tenants are identified by host pid, and gVisor makes that unusually clean. The driver attributes a channel group to the process that opened the device, which for a sandbox is always its Sentry. So every process inside one sandbox, however many the tenant forks, shows up as one pid. The scheduler never has to guess which processes belong together.

The broker is mechanism only. It holds no policy, and it isn't reachable by tenants: it's root-only in the host's /proc, and gVisor's synthetic /proc doesn't expose it. We checked from inside a sandbox, because a sandboxed process often runs as root and could otherwise detach its neighbours. A tenant can still issue the unprivileged timeslice control on its own channel groups through nvproxy, which can clamp it, but it can't issue the privileged restart that would make the change take effect.

The policy: `runsc gpu-scheduler`

The scheduler runs one per node and keeps one planner per GPU. It learns about sandboxes in two ways. When runsc starts a GPU sandbox, it connects to the scheduler and hands the open socket to the Sentry, because the Sentry's seccomp filter forbids it to connect() to anything. That connection carries the sandbox's weight. Separately, runsc announces the sandbox's host pid once the Sentry process exists, which is the first moment the pid is known.

Getting from a working mechanism to an automatic one took three more fixes, each hiding the next:

The pid announcement could be lost. It was a single message with no retry, and the scheduler forgot it if the weight connection dropped. It now retries, and the pid is kept as a property of the sandbox.
The plan has to be re-asserted. A plan written once, a few seconds into a pod's life, landed before the tenant had created its cuBLAS channel groups, so the real ones appeared later at the driver default. The scheduler now re-asserts the full plan every ~2 s, binding any channel group created since.
The gate has to stand down. If the Sentry's gate revokes a sandbox's mappings, the sandbox stops submitting, its GPFIFO drains, and the driver reports it idle, which disturbs the division. With the broker running, every sandbox gets a full-period window, so the gate never revokes and the runlist alone does the dividing.

The first policy that worked set each tenant's timeslice proportional to its weight. On an A100, fully automatic, two cuBLAS pods weighted 3 : 1 divided 1067 : 344 (3.10 : 1) at the same aggregate throughput as no enforcement. For a day, we thought we were done.

The packing attack

The red team broke it by forking. A timeslice binds a channel group, and a tenant owns as many as it creates: each CUDA context creates several, and a tenant that forks four processes creates four times as many. With the share encoded as a per-group timeslice, a tenant's share tracked (channel groups × timeslice). On an A6000, a tenant granted a quarter of the GPU ran four processes in its own pod and took more than a neighbour weighted three times higher: 0.78 : 1 against the requested 1 : 3. Memory and containment held; the share didn't.

The obvious fix, dividing a tenant's timeslice among its channel groups, restored the ratio but cost 42 % of aggregate throughput, because smaller timeslices multiply context switches.

The fix that worked is credit accounting per tenant, the scheme described in the GVM paper ("OS-Level GPU Virtualization", Berkeley/UCLA). It is in pkg/gpusched/credit.go:

Every channel group gets the same large quantum, 4 ms. The quantum only sets how often the GPU switches, not who gets what.
Weight sets how fast a tenant accrues credit. Each tick, the time a tenant consumed is charged back against its credit.
The charge is by channel-group share, read from the broker's tsgs field. A tenant holding n of the N attached groups really does take n/N of the device, so that's what it's charged. Charging every tenant the same would under-charge exactly the attacker this exists to stop.
A tenant that overdraws is detached, all its groups together, and reattached when it recovers. Forking can't help: fifteen channel groups drain one credit pool fifteen times faster.

Two details turned out to be load-bearing. A dead band is required: detaching at zero credit makes tenants that are getting exactly their share flap on and off every tick. And a detached tenant still counts as wanting to run. Once detached, its GPFIFO stops advancing and the driver reports it idle forever, so the scheduler, not the activity signal, has to remember it.

	Weight 75 tenant	Weight 25 tenant	Ratio
A6000, timeslice scheme, honest	126.8	40.7	3.12 : 1
A6000, timeslice scheme, weight-25 tenant forks 4	70.5	90.9	0.78 : 1
A6000, credit, honest	100.9	35.3	2.86 : 1
A6000, credit, weight-25 tenant forks 4	93.9	41.4	2.27 : 1
RTX 5070, credit, honest	131.4	40.0	3.29 : 1
RTX 5070, credit, weight-25 tenant forks 4 (3 → 12 TSGs)	125.7	45.7	2.75 : 1

All figures are cuBLAS SGEMM matmul/s, counted as completed work over the run window. The packer keeps a small edge, 31 % of the A6000 and 27 % of the 5070 against a granted 25 %, so this is blunted rather than perfect. The edge moves inversely to the attacker's channel-group advantage, which tells us it isn't under-charging. It tracks how much spare capacity the device has, which brings us to cost.

The same result held with real tenants. Four tenants, each in its own vCluster on one RTX 5070, three serving vLLM (Qwen2.5-0.5B) under load at weights 100 / 50 / 25 and one adversary: the honest split was 4.25 : 2.12 : 1 against a nominal 4 : 2 : 1, and when the adversary went from one process to four (3 → 12 channel groups), its throughput went from 31 to 27 matmul/s.

What enforcement costs

Enforcement has no fixed cost. What it can give up is an overlap bonus, which only exists when the GPU has slack.

	RTX 5070	RTX A6000
one tenant, no scheduler	171.4	139.4
one tenant, credit scheduler	173.3 (no measurable cost)	139.3 (99.9 %)
two tenants, no enforcement	—	167.5 (120 % of solo)
two tenants, credit scheduler	171.4 (= solo)	136.2 (98 % of solo)

A lone tenant pays nothing. On the 5070, one cuBLAS stream already saturates the card, so two enforced tenants together get exactly what one gets alone. On the A6000, one stream doesn't saturate it, so two unenforced tenants interleave and fill each other's gaps, beating a lone tenant by 20 %. Enforcing a share serialises them and gives that bonus up. We first reported the A6000 figure as 20 % scheduler overhead, which it isn't. Measuring the single-tenant baseline, the run everyone skips, is what told the two apart.

In Kubernetes

None of this helps in a cluster unless something turns what a pod asks for into what runsc enforces. HAMi's scheduler, device plugin and webhook run unchanged: they place pods and keep the books. Its limiter is no longer preloaded into containers. An in-tree mutating webhook restates each pod's admitted request as Sentry flags: nvidia.com/gpumem becomes a memory limit (per device, for multi-GPU pods), and nvidia.com/gpucores becomes a scheduler weight.

Two properties matter when the pod spec is written by the tenant:

Narrow-only. The value configured on the runtime is a ceiling that a pod annotation can lower but never raise, and the webhook overwrites any self-written annotation above what the pod requested. A tenant who requested 1024 MiB and annotated itself 40 GiB saw 1024 MiB. On the B300, a tenant that asked for the host's runc runtime and annotated 256 GiB at maximum weight ran under gVisor at the 20000 MiB and weight 20 it had requested.
Fail closed. The webhook runs with failurePolicy: Fail, and a GPU sandbox that can't reach the scheduler doesn't start. An unmutated or unscheduled pod would run at the whole-device ceiling, so failing open would be a quota escape.

We run each tenant in its own vCluster, as cluster-admin of a virtual cluster with every control applied on the host side; the runtime class is forced to gVisor whatever the pod asks for. That layer is a separate post.

Eight B300s

The B300 node (eight GPUs on NVSwitch) was where we met multi-GPU, and its bugs were the most instructive in the project, because three of them failed open, silently.

The broker's table never freed entries. Every channel group took a slot, and a slot was never released. An 8-GPU NCCL pod takes about 110, so after ten such sandboxes the table was full, new tenants weren't tracked, and a 75 / 25 pair split 645 : 644 with nothing logged.
Every broker command went to one GPU, whichever had last registered a group. 147 of 168 timeslice calls on an 8-GPU pod failed with 0x57, and a tenant opening a context on one GPU could take enforcement away from a pair on another.
One credit pool served the whole node, so sandboxes on different GPUs competed for credit. A weight-25 pod alone on its GPU got 894 TFLOPS instead of ~1330.

All three are fixed: slots are freed at channel-group teardown, each group carries its own GPU, and the scheduler keeps one planner per GPU. That left the obvious follow-up attack: fill the table with idle contexts, and new work goes untracked. A weight-25 tenant took 814 TFLOPS against its weight-75 neighbour's 371 that way. Now a channel group is recorded before the firmware enables it, a group that can't be recorded is refused once a scheduler is running, and one tenant can hold at most a quarter of the table.

With that in place, from inside a gVisor sandbox:

Test	Result
8-GPU pod	10.86 PFLOPS, ~718 GiB/s to every peer, NCCL all-reduce 696 GB/s bus bandwidth
4-GPU pod, from a tenant	all-pairs P2P, ~715 GiB/s NVLink copy, NCCL 624 GB/s
75 / 25 on one GPU, cuBLAS, from a tenant	2.80 : 1
weight-25 pod alone on its GPU, beside a 75 / 25 pair	1332 TFLOPS
weight-25 pod on two GPUs, one shared with a weight-75 pod	1313 TFLOPS on its own GPU, 329 : 1022 on the shared one

nvidia-smi topo -m also needed fixing: NVML places each GPU by walking /sys/bus/pci/devices, which a sandbox didn't have. gVisor now gives nvproxy sandboxes a read-only snapshot of the GPUs' PCI topology, so NCCL sees the NVLink fabric.

What doesn't work yet

A patched driver. The broker is host kernel code in our fork of the open modules. Memory quotas work on a stock driver; compute shares don't.
The scheduler is a single point of failure. It fails closed, on purpose: with it down, GPU pods don't start.
Long kernels overshoot. A GPU can't be made to abandon work already submitted. A kernel longer than its turn runs to completion, and credit charge-back recovers the time afterwards instead of preventing it.
The packing edge is blunted, not gone, at 2 to 6 points above the granted share in our measurements.
Spatial partitioning is a ceiling, not concurrency. The TPC partition control works once issued from the right place (13, 18 and 24 of the 5070's 24 TPCs give 272, 372 and 458 matmul/s), but two tenants on disjoint halves perform the same as two on the same half, because separate contexts still take turns on the engine.
Memory accounting gaps. Unified memory is charged at reservation, and fabric memory and EGM aren't accounted yet.
Telemetry. A sandbox can still read the GPU's live power draw, a low-bandwidth signal of a neighbour's activity.
No checkpoint/restore of a CUDA sandbox: several object types every CUDA context creates don't implement restore in nvproxy.
Each GPU gets measured. The runlist path is confirmed on GB205 (RTX 5070), GA102 (RTX A6000), GA100 (A100) and the B300. We no longer predict a new die from its class; that is the mistake that cost us the most.

What we'd tell ourselves two months ago

Work submitted to a GPU can't be intercepted, and we treated that as a limit on compute isolation. It's a limit on interception. The GPU will divide its own time between tenants if you ask it correctly, and "correctly" was the entire difficulty: the right control, on a live object, followed by the call that makes the firmware act. For several days, three small mistakes about how to ask looked exactly like a fact about what's possible. What settled it wasn't a better argument. It was reading one status code correctly and moving one call to a later point in the driver.

The other lesson is to test like the tenant is lying to you. Graph replay, process packing and a silently full table were each invisible to a benchmark that behaved itself, and each was found by someone trying to cheat.

The design overview is GPU-ISOLATION.md, the driver side is DRIVER-CHANGES.md, and the red-team record is SECURITY-FINDINGS.md. AMD turned out to be a mirror image, with both kinds of division reachable from the Sentry on a stock driver. That's another post.

Conformant Is Not Performant: What OpenFlow Can Teach O-RAN

2026-10-29T00:00:00Z

If you worked in networking around 2010, you remember the excitement around OpenFlow. Pull the control plane out of the switch, put it on a server, speak a standard protocol between the two, and the network becomes software. Vendor lock-in would end. Researchers and operators would invent new things without waiting on a vendor's release schedule.

Some of that came true. Most of it didn't happen the way anyone expected.

If you work in wireless today, the story of O-RAN should sound familiar. Open the interfaces between the pieces of the radio access network. Let operators mix and match vendors. Add a RAN Intelligent Controller (RIC) so third parties can write apps that tune the network. It's the same promise, and it's running into some of the same walls.

We spent more than a decade in data center SDN before turning to 5G, so we have watched one industry go through this already. This post compares the two journeys: what each set out to do, where each got stuck, and what the eventual success in data center networking does and doesn't tell us about where wireless should go next.

Same goal, different decade

OpenFlow and O-RAN were both reactions to the same problem: a vertically integrated box whose internals the buyer couldn't see or change. In the data center it was the switch: ASIC, operating system, and protocols from one vendor. In wireless it's the base station: radio, baseband, and software from one vendor.

Both proposed the same cure: define open interfaces between the pieces, and let an ecosystem grow around them. OpenFlow separated the control plane from the forwarding plane and defined a protocol between them. O-RAN split the RAN into a radio unit (RU), distributed unit (DU), and central unit (CU), defined interfaces between them, and added a RIC with its own interfaces for third-party applications.

Both changed what buyers believed was possible. Both also standardized things that turned out to be much harder to make interoperable in practice than on paper.

What happened to OpenFlow

OpenFlow's core bet was that a switch could be reduced to a generic match-action table, programmed by a controller over a standard protocol.

The problem wasn't the protocol. It was the gap between what the specification allowed and what implementations actually did. Real switching ASICs have fixed pipelines with many tables of different sizes and capabilities. OpenFlow 1.0 assumed one flexible table; later versions added multiple tables, but couldn't make every ASIC look the same. Much of the spec was optional. Vendors shipped switches that were OpenFlow-conformant but supported a different subset of matches and actions, had flow tables a fraction of the size you needed, or quietly sent some rules to a slow software path. OpenFlow 1.3 became the version everyone used; later versions added features few switches implemented. Efforts like Table Type Patterns and Broadcom's OF-DPA tried to describe what a given switch could actually do, which was an admission that "supports OpenFlow" didn't mean much on its own.

So for a controller application to work, you had to find the intersection of what your controller used and what your particular switch got right. That intersection was often small.

Where OpenFlow did work, one party controlled both ends. Google's B4 WAN used it successfully, with switches and controllers Google built itself. Open vSwitch, a software switch, made OpenFlow the basis of network virtualization. In neither case was there a multi-vendor interoperability problem to solve.

The hyperscalers ended up getting programmability a different way: BGP in the data center, running on switches whose software they controlled. That brings us to SONiC. But first, O-RAN.

O-RAN's report card

Open fronthaul: real success, real growing pains

The O-RAN open fronthaul (OFH) interface between the DU and the radio unit, the 7.2x split, is the clearest success of the O-RAN program. Before it, the link between baseband and radio was a vendor-specific variant of CPRI, and you bought both ends from the same vendor. Today there's a genuine market of O-RUs from many suppliers, and open-source DUs can drive them. That's a structural change.

But it isn't plug-and-play yet. OFH leaves a lot of room for choices: compression schemes, section types and extensions, beamforming methods, timing windows, synchronization configurations. Two products can each pass conformance testing and still not work well together, or work but deliver a fraction of the expected throughput. As a wise colleague in the OCUDU community puts it, "conformant is not necessarily performant."

That is exactly the OpenFlow experience. A switch that implements the spec correctly but has a 2,000-entry flow table is conformant. An O-RU that accepts every message correctly but needs hand-tuned timing offsets and one particular compression mode to reach line rate is conformant, too. Neither is what you were hoping to buy.

OFH is getting better, through interoperability profiles, plugfests, and hard-won integration experience. It's improving because the interface is narrow and the incentive is strong: nobody can sell an O-RU that doesn't work with the DUs people are actually deploying.

E2 and the RIC: the OpenFlow of O-RAN

The near-real-time RIC is a great idea. Exposing RAN telemetry and control to third-party applications is how wireless gets the same innovation curve that data center networking got from programmability. Traffic steering, slicing, energy savings, interference management, mobility optimization: there's a long list of valuable applications that fit comfortably within the RIC's 10-millisecond-to-1-second control loop.

The interface that makes it work, E2, is the most unevenly supported interface in O-RAN, and for the same reason OpenFlow struggled. E2 itself (E2AP) is a fairly thin transport. The meaning lives in the service models: E2SM-KPM for performance metrics, E2SM-RC for RAN control, and others. Those service models are where implementations diverge:

KPM is fairly widely implemented, but which metrics a given RAN actually reports, and at what granularity, varies.
RC is implemented partially, with each RAN supporting a different subset of control styles and actions.
Other service models are rare outside specific vendor and research combinations.

So "my RAN supports E2" is a lot like "my switch supports OpenFlow." It tells you the protocol will connect. It doesn't tell you whether the xApp you want to run will work. An xApp developer is back to finding the intersection of what their app needs and what a particular RAN implements, and that intersection is often small.

The cost of implementing service models properly is a big part of why. They're defined in ASN.1, versioned independently, and tied to large specifications. Adding a new metric or control means writing spec-compliant ASN.1, generating code for it, and mapping it carefully into the RAN's internals. That's a lot of work for a RAN developer whose users aren't asking for it yet, and it's a chicken-and-egg problem: there are few xApps because support is spotty, and support is spotty because there are few xApps.

The tell: everyone builds around E2

Here's the clearest sign: people who want to do real work with a RAN often skip E2 entirely. Project after project has added its own interface directly into the DU: ZeroMQ sockets, WebSocket APIs, shared memory queues, custom gRPC services. It is simply easier to add the hook you need, in the format you want, than to define and implement a conformant service model.

Some of this work also targets applications that really do need to run faster than E2 allows, like per-slot scheduling decisions or processing I/Q samples and channel estimates that are too large to ship across a network. Microsoft Research's Janus work (now the open-source jbpf framework) loads small, verified eBPF "codelets" directly into the RAN's hot path. Northeastern University's dApps work defines an E3 interface for applications that run right next to the DU. These are valuable additions. But the main reason people go around E2 isn't timing. It's the cost of conformance.

We've seen this before, too. In the SDN era, operators who wanted programmability without OpenFlow's pain used NETCONF, OVSDB, BGP with custom route injection, or simply ran their own agents on the switch. The demand for a programmable network was real. It just found a different path.

Why SONiC succeeded where OpenFlow didn't

OpenFlow didn't deliver an open data center network. SONiC did, along with the hardware ecosystem around it. Microsoft built SONiC for Azure, open-sourced it in 2016, and moved it to the Linux Foundation in 2022. It now runs on switches from many vendors and ASIC families, on millions of ports. A few choices made the difference.

It standardized the inside of the box, not the wire. Instead of a remote protocol every switch had to speak, SONiC relied on the Switch Abstraction Interface (SAI), a C API defined at the Open Compute Project that each ASIC vendor implements once for its chip. The interoperability boundary became a local driver API, where chip differences are easier to handle.

It reused protocols that already worked. SONiC didn't replace BGP with a controller. It ran BGP (through Quagga, later FRR) on the switch. Centralized intelligence came in through automation and policy that configured those protocols, not by replacing them.

The code came first. SONiC was a working production system before it was a community standard. "It runs SONiC" became the practical definition of interoperability. Features were added when a real operator needed them.

It had an anchor customer running it in production. Azure ran SONiC at scale. That gave vendors a reason to write SAI drivers and gave the code a level of hardening no lab could.

Its components share a common state store. Inside SONiC, every component, from the BGP daemon to the process that programs the ASIC, communicates through a set of Redis databases: configuration, application state, ASIC state, and operational state. Components don't call each other; they write the state they own and subscribe to the state they need. Adding a new feature means adding a container that reads and writes those tables. This is the officially sanctioned version of the "hack your own interface" that people are doing to the DU today.

ONIE separated buying hardware from choosing software. The Open Network Install Environment, contributed by Cumulus Networks to OCP, gave bare-metal switches a standard way to boot and install a network operating system.

The result is that today you can buy a white-box switch, install SONiC, run real routing protocols on it, and add your own software alongside them. That's what OpenFlow promised. It just arrived by a different route.

Where the analogy holds, and where it breaks

It's tempting to map SONiC's pieces one-to-one onto wireless. Some of them map well. Some don't, and the places where they don't are the most interesting part.

OCUDU is in SONiC's position

The strongest parallel is the software. OCUDU is an open CU/DU stack with government backing, hosted at the Linux Foundation, with operators, chipmakers, and equipment vendors as members. If it becomes the stack that real deployments run, "it runs OCUDU" can become the practical definition of interoperability, the way "it runs SONiC" did. That's the code-first path, and it's the one that worked.

There's no single SAI, because the PHY is too big

SAI worked because a switching ASIC is a well-bounded thing. It does packet forwarding at line rate, in fixed-function hardware, and the operating system programs it through tables. The boundary between "commodity silicon" and "software" is obvious.

The RAN's physical layer is a different animal. It's the bulk of the compute, it's algorithmically rich, and the quality of the receiver is where a lot of real differentiation lives. And OCUDU implements the PHY itself, in software, with accelerators where they help. So the obvious candidate for "the SAI of wireless," the Small Cell Forum's FAPI interface between the PHY and the layer above it, doesn't fit. FAPI is the right boundary for merchant-PHY silicon that ships a complete PHY, but in an OCUDU deployment the PHY is part of the open stack, not under it.

If the PHY is in software, where is the hardware boundary? There are really three:

Lookaside accelerators for functions like LDPC decoding, reached through driver frameworks like DPDK's bbdev and the O-RAN acceleration abstraction layer. This looks less like SAI and more like the driver model of a software router running on servers.
The fronthaul to the radio unit. With the 7.2x split, the RU does the low PHY (FFT, beamforming, digital front end), and the DU programs it every slot through control-plane messages. With split 8, the RU is essentially converters and RF, and the DU does everything else.
The compute platform itself: CPUs, GPUs, and the timing hardware that keeps it all synchronized.

Of these, the radio unit is the closest thing to a switching ASIC: fixed function, line rate, built on vendor silicon. That makes the RU interface the truest analogue of SAI. In fact, OCUDU already has an internal radio-unit abstraction with two kinds of backends: open fronthaul for 7.2x O-RUs, and SDR drivers for split-8 radios. That internal abstraction is arguably the SAI of the open RAN, with 7.2x and split 8 as its "drivers."

But the analogy breaks in an important way. SAI is a local API whose driver is written by the silicon vendor. The 7.2x boundary is a wire protocol with tight timing, and on the other side is RU firmware the operator can't see or change. That's why "conformant is not performant" bites so hard: you can't fix or even inspect the other half.

The biggest gap: the radio unit is still mostly closed

Data center networking succeeded with white-box switches because a merchant silicon vendor shipped the ASIC, the SDK, and the SAI driver, and a white-box vendor built the board around it. Anyone could then run SONiC.

Radio units are only starting down that road. Take an RU built on the NXP LA12xx platform. NXP provides the programmable baseband silicon and a low-level SDK, not a complete PHY with a standard interface on top, open or closed. Building the low PHY, the RU side of the 7.2x protocol, the management plane, and the timing integration is left to the RU maker. We're not picking on NXP; this is the state of the layer across the industry. The "driver" for an RU isn't a thin shim. It's an entire low-PHY implementation.

The most serious effort to change this has come, fittingly, from the Open Compute Project, the same community that gave data center networking SAI and ONIE. Meta's Evenstar program built a modular reference design for a radio unit, with Jabil manufacturing commercial units from it, and in 2023 Meta moved the work to OCP. The contribution included the hardware specification and design files, DPD and CFR firmware, and test automation, and Meta contributed its Open M-Plane management software to the Linux Foundation. The design is built so that power amplifiers, filters, and even the radio silicon can come from different suppliers. That's the white-box idea, applied to radios.

OCP has since folded Evenstar into a broader AI-Native Wireless Access effort. Two of its workstreams stand out. An AI-accelerated radio unit continues the open RU work. And a Radio Abstraction Interface aims to define how third-party applications run on the radio unit itself. Even the name echoes SAI, and it targets exactly the layer this post argues is missing. It's early, with specifications still to come, but it's pointed in the right direction.

Still, wireless today has merchant silicon for radios, an open reference design for the hardware, and no widely available open software stack for the RU. Most O-RUs remain closed boxes at the bottom of an otherwise increasingly open system. That's a bigger gap than anything at the top of the stack, and we think it's where some of the most important work is.

Is E3 the state bus?

The other missing piece is SONiC's shared state store. The open DU has no equivalent, which is why every project invents its own hook.

The leading candidate is E3, and it's moving fast. The O-RAN Alliance published E3 in a research report (not yet a specification), authored by Northeastern, SoftBank, and NVIDIA. DeepSig, one of OCUDU's leads, has released a preview of a dApp platform for OCUDU with its own E3: an in-process C interface that runs in microseconds, plus an E3 management protocol and a data profile for streaming telemetry like PUSCH and SRS data.

It's still early, and it's a real step forward. But is it a SONiC-style state bus? Not yet, and maybe not by design. A few observations:

It's an application interface, not an internal integration fabric. SONiC's databases are how SONiC's own components talk to each other. E3 is how outside applications talk to the DU. DeepSig's design is explicit that dApps don't call each other. The model is subscribe, indicate, and control, the same as E2.
It inherits E2's machinery. The management protocol uses ASN.1 over SCTP and follows E2's setup, subscription, and indication structure. That makes bridging to E2 easy, but it brings along some of the same implementation weight.
There are already two E3s. By DeepSig's own description, its E3 is not a ratified O-RAN interface, and its wire format isn't byte-compatible with the libe3 implementation from the research-report lineage. That's understandable for a preview, but it's exactly how fragmentation starts.
It lives in the reference implementation. This is the big difference from E2. If the dominant open DU ships E3, with versioned schemas and a frozen ABI, then the service model gap may never open in the first place, because there's one implementation everyone uses.

So E3 could go either way. If it becomes "the interface OCUDU ships," with one implementation and a well-defined state model behind it, it gets the code-first advantage SONiC had. If it becomes a family of compatible-in-spirit protocols implemented differently by each RAN, it will have E2's problem at a faster timescale.

What we expect

Here's what we expect over the next several years.

E2 will survive, in a narrower role. It will be used for telemetry and slower policy, with KPM as the most widely used piece, much as OpenFlow survives in software switches and specific deployments. It will matter most where one vendor controls both the RAN and the RIC.
Real programmability will move toward the open DU itself. Extension points inside the open stack, like codelets and dApps, will become the main way to add intelligence, because it's easier to build on one implementation than to wait for many to converge on a spec.
The open stack will become the reference. Interop results, test harnesses, and deployment experience built around OCUDU will matter more than paper conformance.
Open fronthaul will keep maturing through profiles and testing, and will become boring, which is the best thing that can happen to an interface.
The RU will remain the hardest part to open, until someone treats the low PHY and RU software as a product category in its own right.

What we should push for

Code-first interfaces. An interface should be validated by a working open implementation, with tests, before it's finalized. A service model that doesn't exist in the open stack isn't ready.

One state model for the open DU. Rather than each application interface mapping separately into RAN internals, the DU should have one well-defined, versioned model of its own state and controls, the way SONiC has its databases. E2 and E3 can then be views onto that model, and much of each service model could be generated rather than hand-written. That would make conformant implementations far cheaper, and it would make E3 much less likely to repeat E2's story.

Safe extension inside the node. Code running in the hot path of critical infrastructure must not be able to crash it or leak data from it. Sandboxing and verification should be built in from the start, and the interfaces need authentication before they leave localhost. This is where formal methods earn their keep.

An open RU. Open low-PHY software and open RU-side fronthaul and management, on open FPGA and SDR platforms as well as merchant baseband silicon. Split 8 radios are a practical starting point, because the boundary is simple and the low PHY can run in the open stack. OCP's open RU and Radio Abstraction Interface work deserves support from everyone who wants an open RAN, including the open-source CU/DU community. Until the RU's software is as open as its hardware design, the bottom of the open RAN is still a closed box.

Performance, not just conformance. Plugfests and certification for open fronthaul should publish performance results (throughput, latency, timing margins) alongside pass/fail, using open test harnesses anyone can reproduce. "Conformant and performant" should be the bar.

Anchor deployments in production. SONiC succeeded because Azure ran it. Open RAN needs real networks, with real users, running the open stack. Government programs are in a unique position to provide that, and to require openness all the way down as a condition.

The lesson

OpenFlow wasn't wrong about the goal. It was wrong to trust a specification to deliver interoperability that only working code and real deployments could. Data center networking got there in the end by standardizing the inside of the box, reusing what worked, and letting an open implementation become the standard.

Wireless won't follow that path exactly. The PHY is bigger than a switching ASIC, and the radio unit is still closed. But O-RAN has already done the hardest part: it convinced an industry that the RAN should be open. Open fronthaul shows it can work. With OCUDU, wireless now has its SONiC candidate. What it needs next is an internal state model to build on, and an open radio at the bottom of the stack. We'd like to help build both.

A tensor accelerator in PolarFire fabric, 81× over the board's own CPUs

2026-10-30T00:00:00Z

A Microchip PolarFire SoC Discovery Kit has four RISC-V application cores and a large block of FPGA fabric beside them. We put Apache TVM's VTA — an open, parameterised INT8 tensor accelerator — into that fabric, drove it from Linux on the MSS cores, and compiled real ResNet-18 convolution layers onto it through TVM.

The result is worth the fabric it occupies: 49 GOPS across ten convolution layers, 84 % of the configuration's theoretical peak while the array is running, and 81× the throughput of the same layers on all four application cores. This post covers how it is put together, what it measures, and the one PolarFire-specific RTL requirement that a port like this depends on.

49 GOPSten ResNet-18 conv layers, end to end through TVM

81×against the same layers on the board's four RISC-V cores

84 %of theoretical peak while the array is running

91 %of wall time spent inside the accelerator

How it is put together

VTA sits in the fabric behind FIC0, the MSS's AXI port into the fabric, with its control registers on one interconnect and its DMA master on another. The configuration is a 1 × 16 × 16 INT8 array at 125 MHz: 256 multiply-accumulate units, two operations each, giving 64 GOPS of theoretical peak. Synthesis maps the array onto 128 of the MPFS095T's 292 MATH blocks, two multipliers per block in dual 9 × 9 mode, and the scratchpads onto LSRAM.

The software side needs no kernel module. A userspace driver maps VTA's registers and a reserved, non-cached DRAM region through /dev/mem. Using the non-cached alias is what keeps the CPU and the accelerator coherent without any cache maintenance: there is no cached view of those buffers to go stale, so the driver never has to flush or invalidate. The region is reserved no-map in the device tree, so Linux never allocates from it.

On top of that the ordinary TVM path works end to end: write a schedule, vta.build it, cross-compile the host side for riscv64, ship it to the board over RPC, execute on the accelerator, and check the result against a NumPy reference. Two details are specific to this board. The RISC-V target needs both its ISA string and its ABI pinned — the distribution's compiler defaults to an extension set whose vector code will not run on this part, and LLVM otherwise emits soft-float-ABI objects that the lp64d linker refuses to merge with the rootfs libraries. And modules are exported as cross-compiled shared libraries rather than object files for the RPC server to link, which it cannot do here.

What it measures

Every convolution layer of ResNet-18, each compiled twice from the same TVM source — once for the accelerator, once for the board's own CPUs:

Layer	Shape	CPU ms	VTA ms	VTA GOPS	Speed-up
C2	56² 64→64 3×3	286.5	4.75	48.67	60×
C3	56² 64→128 3×3 /2	153.3	2.38	48.61	64×
C4	56² 64→128 1×1 /2	15.7	1.06	12.16	15×
C5	28² 128→128 3×3	294.9	4.47	51.73	66×
C6	28² 128→256 3×3 /2	222.9	2.17	53.31	103×
C7	28² 128→256 1×1 /2	25.6	0.74	17.26	34×
C8	14² 256→256 3×3	448.4	4.15	55.70	108×
C9	14² 256→512 3×3 /2	230.5	2.08	55.52	111×
C10	14² 256→512 1×1 /2	25.4	0.69	18.66	37×
C11	7² 512→512 3×3	459.8	4.10	56.45	112×
Total	1310 MOP	2163	26.6	49.28	81×

Two things keep that table honest. The CPU baseline uses all four cores — measured at 4.09 cores across four threads, by sampling /proc/<pid>/stat deltas rather than trusting ps, whose %CPU is a lifetime average and reads near zero on a busy worker. And neither side is autotuned: TVM reports a fallback schedule for both, because tophub carries no entries for this board. So 81× is the right order of magnitude rather than a precise figure, and tuning would likely help the CPU proportionally more, since the accelerator is already near its ceiling and the CPU is nowhere near its own.

The CPU is flat at 0.5–0.8 GOPS whatever the layer: it issues scalar INT8 multiply-accumulates, so its time tracks op count and nothing else. VTA ranges from 12 to 56 GOPS depending on how well the shape suits the array. The gap widens with channel depth — 60× at 64 channels, 112× at 512 — because deeper channels fill a 16 × 16 array better and the CPU gains nothing from them. The 1 × 1 layers are the array's weakest case: they move nearly as much data for a ninth of the arithmetic, so they are bound by DMA rather than by compute. Note that the CPU holds its usual rate on those same layers, so the narrower gap there is the accelerator slowing down, not the CPU catching up.

Reading VTA's own cycle counter alongside wall time separates accelerator time from system time. Of 26.6 ms, the array is busy for 24.3 ms: a 91 % duty cycle, 53.9 GOPS while running, 84 % of peak, rising to 96 % on C11. MAC utilisation climbs as spatial size shrinks and channels grow, from 83 % on C2 to 96 % on C11, because fewer and larger tiles mean fewer instructions issued per unit of arithmetic.

Instruction queues must be synchronous-read on PolarFire

One RTL change is a hard requirement for this port, and it is not obvious from the Chisel.

VTA's LOAD and STORE modules each buffer their instruction stream in a queue. Upstream both declare a plain Chisel Queue, which is an asynchronous-read Mem. At 512 entries × 128 bits, PolarFire synthesis maps that into LSRAM — which can only read synchronously. The instruction presented at the queue's output then does not match what was written into it. COMPUTE, whose micro-op and accumulator loads are driven the same way, has always used SyncQueue and is unaffected.

// Load.scala and Store.scala
- val inst_q = Module(new Queue(UInt(INST_BITS.W), p(CoreKey).instQueueEntries))
+ val inst_q = Module(new SyncQueue(UInt(INST_BITS.W), p(CoreKey).instQueueEntries))

The corruption is visible directly in hardware. Adding read-only counters to VTA's control register block, tapping the instruction at the queue's input and again at the moment the tensor load starts, the two disagree:

Fetch → LOAD queue   LOAD inp  xsize=2  ysize=1  dram=205520912   written
tensor-load start    LOAD inp  xsize=2  ysize=0  dram=205520896   read back

What a corrupted instruction field does downstream

Corrupt xsize to 0 and the instruction decodes as a sync: the load is skipped silently, no input data reaches the MAC array, and every product becomes a multiplication by zero. The weight load in the following instruction is untouched, so weights look perfectly healthy.

Corrupt ysize to 0 and the DMA command generator finishes at line index ysize-1, which wraps to 65535. It issues no command at all and never completes.

Which bits corrupt varies between builds, so the same root cause presents as wrong results, as a stall, or intermittently as neither.

With both queues on SyncQueue the full suite passes at every size, sixty consecutive executions are correct, and timing improves as a side effect — worst accelerator slack goes to +0.980 ns — because SyncQueue maps onto LSRAM cleanly instead of fighting the inference.

The portable lesson is worth stating plainly, because upstream VTA runs this same code on Zynq without trouble: an asynchronous-read Mem is fine in Xilinx distributed RAM and silently wrong in PolarFire LSRAM. Nothing in the Chisel marks the difference, and synthesis does not warn.

What's left

The accelerator is now the limit rather than the software around it, which sets the order of what to do next.

Raise the clock. The build closes at 125 MHz with +0.980 ns of slack and a minimum period of 6.885 ns, implying roughly 145 MHz of headroom. Peak scales linearly and the array already runs at 84 % of peak, so this converts almost directly into throughput. It requires the MSS FIC clock configuration to match the fabric clock.

Run a whole network. Compiling ResNet-18 as a single Relay graph — quantise, pack to the accelerator's tensor layout, build, run under the graph executor — keeps weights resident and intermediates on the device, and turns ten measured layers into an actual classifier. The first thing to measure there is the CPU-side share: the 7 × 7 × 3 first layer, pooling and softmax stay on the application cores, and on an in-order core those could cost more than the 24 ms the accelerator spends on everything else. The CPU column above makes that predictable rather than a surprise.

Autotuning needs an upstream fix first, and is worth less than it sounds. VTA's schedule_conv2d_packed assumes its output is an epilogue operation, so when autotvm builds the configuration space the output is the convolution stage and gets scheduled twice. Beyond that, the payoff is capped: the 3 × 3 layers already run at 81–96 % MAC utilisation, and the 1 × 1 layers where the headroom actually sits account for 2.5 ms of 26.6 ms.

A wider array has the largest ceiling. The design uses 128 of the part's 292 MATH blocks; raising the block or batch factor raises peak proportionally, bounded by scratchpad LSRAM capacity and by timing — the path from the input scratchpad to the multiplier inputs is the one that tightens first.

The RTL, driver, test harnesses, debug-register decoder and the full Libero build flow are on the polarfire-timing branch of our tvm-vta fork; the TVM-side target support is on polarfire-mpfs.