At Hot Chips 2026, the industry converged on a new bottleneck: AI is moving from one-shot answers to agents that reason, call tools, and keep working. That means far more generated tokens, and far more time spent in decode.
TL;DR
- Agentic AI generates more tokens and spends most of inference in memory-bound decode, so system bandwidth matters more than peak compute.
- Across six frontier-model configs SambaNova analyzed, decode was 75% to 97% of model-inference time.
- Model Bandwidth Utilization (MBU) measures how much installed bandwidth actually moves weights and KV-cache: installed bandwidth times MBU equals tokens per second.
- The SN50 RDU pairs a dataflow execution model and 432 MB on-chip SRAM with scale-up and scale-out Ethernet to keep bandwidth productive across chips.
- Scaling modeled DeepSeek-R1 from 64 to 256 RDUs lifts per-user speed from 200 to 500 tokens per second (TPS) while MBU holds between 44% and 51%.
That shift makes the hardware challenge much more interesting. The next generation of inference will not be won with one big chip. Frontier models already span multiple chips, and their speed depends on how efficiently processors, memory, and networks operate as one system. A fast chip matters. A fast system matters more.
Our Hot Chips talk, “Dataflow at Scale: the SN50 RDU,” focused on that full-system problem: How much installed bandwidth reaches the model; how dataflow keeps work moving inside the RDU; and how the network preserves that efficiency as models scale across chips, nodes, and racks.
Raghu Prabhakar, Chief Architect at SambaNova, presents the SN50 RDU at Hot Chips 2026.
From Peak Bandwidth to Useful Model Bandwidth
Previous SambaNova blogs explain the decode bottleneck and why disaggregated inference maps compute-heavy prefill to GPUs and memory-bound decode to RDUs. Our talk started with the metric that connects that architecture to delivered token speed: Model Bandwidth Utilization (MBU).
This idea is well-established, not new. The MosaicML engineering team at Databricks introduced MBU as achieved model-data bandwidth divided by a system’s peak memory bandwidth. Peak high bandwidth memory (HBM) tells you what the hardware could move. MBU tells you how much of that bandwidth is actually moving model weights and key-value (KV) cache data during token generation. For bandwidth-bound decode, installed bandwidth multiplied by MBU gives effective model bandwidth, which is the part that translates into tokens per second.
In the six frontier-model configurations SambaNova analyzed, decode represented 75% to 97% of model-inference time. At the high end, the DeepSeek-V3 case (8K input, 1K output, B300 FP4, and one active request) spent 97% of model-inference time in decode. The exact split varies by model, context, precision, and concurrency, but the system lesson is consistent: Peak compute alone does not predict the speed users will feel from an agent.
MBU connects peak HBM bandwidth to the model-data movement that drives token generation.
Source: Raghu Prabhakar, Hot Chips 2026. Concept reference: MosaicML/Databricks.
This distinction gets more important as systems grow. Adding accelerators increases theoretical bandwidth, but per-user token speed will not rise with it if synchronization, memory traffic, or communication overhead drives utilization down. The goal is not simply more bandwidth. The goal is more bandwidth that the model can use.
How Dataflow Turns Bandwidth into Tokens
The SN50 RDU starts with a dataflow execution model rather than a sequence of separately launched kernels. The compiler maps the model across a tiled mesh of compute and memory units. SN50’s 432 MB of distributed on-chip SRAM keeps frequently used data close to compute and allows intermediate values to flow between operations without repeated trips to HBM.
Persistent decoder execution and operator fusion keep the model moving across token steps instead of repeatedly stopping and resuming work at kernel boundaries. Double-buffered memory units begin moving the next model tile while the current computation runs. Collective communication can terminate in SRAM, avoiding unnecessary HBM traffic and allowing compute, memory access, and communication to overlap.
That overlap is the link between MBU and scale. Useful bandwidth is not a property of memory alone; it is the result of keeping the entire execution pipeline productive. Once a model spans multiple RDUs, the network becomes part of that pipeline.
Inference at Scale Needs Scale-Up and Scale-Out
Dense frontier models and mixture-of-experts models span many accelerators. Their fabrics must carry frequent collectives and dynamic token movement without turning communication into the next bottleneck.
The SambaNova SN50 uses three connected network domains. Inside an eight-socket node, seven integrated 800G Ethernet links form a fully connected topology, which gives each RDU a direct path to other RDUs in the node. Beyond the node, two additional 800G ports per RDU connect to the inter-node scale-up fabric. Across those domains, the SN50 integrates ten 800G Ethernet ports with an aggregate 2 TB/s per RDU. In the 64-socket example, two 64-port 800G switches connect every node to both switches, with one link from every RDU going to each switch.
Scale-out has a different job. Each SN50 RDU uses a 400G RoCEv2 NIC to connect scale-up domains across a rail-optimized Ethernet network. This layer supports disaggregated inference and KV-cache transfer between scale-up domains. A separate front-end data center network remains available for hosts, storage, and management.
SN50 combines a fully connected eight-socket node, 800G inter-node scale-up, and a 400G RoCEv2 scale-out path between scale-up domains. Source: Raghu Prabhakar, Hot Chips 2026.
The separation matters because the traffic is different. Scale-up carries the tightly coupled communication required to split one model across RDUs. Scale-out connects larger domains and heterogeneous inference stages. Each tier is built around the communication pattern it has to sustain.
Why Tensor and Expert Parallelism Need Different Network Behavior
Tensor parallelism splits large tensor operations across RDUs. Every step depends on collectives, such as reduce-scatter, all-gather, and all-reduce, so communication has to keep pace with matrix computation. The on-silicon tensor-parallel general matrix multiplication (GEMM) benchmark presented at Hot Chips achieved at least 70% TFLOPs utilization, the share of available floating-point compute kept productive, across 8, 16, and 32 SN50 sockets. At 32 sockets, the benchmark achieved full overlap of compute and communication. This is a scoped kernel result, not end-to-end model performance, but it shows the behavior the scale-up fabric is designed to preserve.
SN50’s on-silicon tensor-parallel GEMM benchmark achieved at least 70% TFLOPs utilization across 8, 16, and 32 sockets, with full compute-communication overlap at 32 sockets. Source: SambaNova on-silicon measurement presented by Raghu Prabhakar, Hot Chips 2026, slide 28.
Expert parallelism creates a different challenge. In a mixture-of-experts model, the router selects different experts for different tokens. Dispatch and combine therefore generate dynamic traffic whose destinations change with every batch. SN50 supports both all-to-all and broadcast-and-filter approaches. All-to-all filters at the source and avoids extra traffic, but produces non-uniform flows. Broadcast-and-filter creates more uniform traffic and can begin alongside router computation, at the cost of sending additional data.
In SambaNova’s 64-socket expert-parallel configuration, more than 70% of the bandwidth was used to load experts. Across both tensor and expert parallelism, bandwidth becomes performance when the execution model can overlap communication and match the network behavior to the traffic the model actually generates.
Keeping MBU High as the System Grows
The scaling question is simple: Can more chips deliver more speed without giving back the useful bandwidth that made each chip fast?
Across SambaNova’s modeled DeepSeek-R1 8K-input/1K-output operating points, projected per-user speed rises from 200 to 300 to 500 tokens per second as the configuration scales from 64 to 128 to 256 SN50 RDUs. MBU remains at 51%, 44%, and 45%, respectively. These are modeled operating points, not measured production results, and outcomes will vary by workload. They illustrate the design target: Increase decode capacity while preserving a substantial share of useful model bandwidth.
More RDUs increase aggregate bandwidth, while dataflow with the scale-up fabric are designed to keep enough of that bandwidth productive for token speed to climb. Scaling is not only about fitting a larger model, it is about keeping the model moving.
The Hot Chips Takeaway
The takeaway from Hot Chips 2026 is straightforward: Agentic inference is a systems problem. GPUs remain a strong fit for training and compute-bound prefill. RDUs are purpose-built for memory-bound decode. CPUs coordinate tools, APIs, vector databases, and actions around the model. The network lets many processors operate as one inference platform.
SambaNova has already shown what that hybrid architecture can deliver. At RAISE Summit 2026, a preview system using four NVIDIA H200 GPUs for prefill and 16 SN50 RDUs for decode reached 763 output tokens per second on MiniMax M2.7 with 10,000 input tokens in benchmarking by Artificial Analysis. SemiAnalysis later benchmarked SambaRack SN50 on the same model in two tensor-parallel configurations. Those results are covered in the RAISE MiniMax blog post and SemiAnalysis benchmark recap.
Peak bandwidth describes possibility. MBU shows how much of that possibility reaches the model. Our Dataflow Architecture keeps model data moving inside the RDU. Scale-up Ethernet carries tensor- and expert-parallel traffic; scale-out Ethernet carries KV-cache transfers and disaggregated inference traffic across domains. Put those pieces together, and the architecture is designed to give agentic AI fast tokens, while preserving useful bandwidth as models scale.