Every day, the AI ecosystem is evolving and pushing for further optimizations from chips to models. Why? Because coding and enterprise agents are delivering real productivity gains to people today in tools like OpenClaw, but they are taking hours – sometimes days – to complete due to the size of these models and the long reasoning chains required to deliver accurate results.
At the NVIDIA GTC 2026 keynote, a chart was shown with System Throughput on the Y-axis and Speed on the X-axis. At SambaNova, we agree that the future of AI hardware will boil down to this representation. This is, in fact, the same way we framed the launch of our fifth-generation SN50 RDU chips.
All agents want faster performance, but that speed needs to be served in a token-efficient way that enables inference providers to concurrently support requests from many simultaneous agents. The key is delivering agentic inference in the Goldilocks Zone.
What Is Hybrid AI Architecture?
Hybrid AI architecture is an infrastructure design that combines different types of hardware, such as GPUs and RDUs, to optimize each stage of AI workloads, particularly for large-scale inference.
Rather than relying on a single system, hybrid architectures assign tasks based on their computational requirements. GPUs handle compute-intensive operations like training and prefill, while specialized processors like RDUs are optimized for memory-intensive tasks such as decode. This separation allows each component to operate more efficiently.
In the context of agentic AI, where models generate long reasoning chains and serve multiple concurrent requests, this approach is critical. Hybrid AI architectures enable faster response times, higher throughput, and better resource utilization by matching the right hardware to the right workload.
What Are the Key Metrics that Matter for Hybrid AI Architecture?
To evaluate the performance of a hybrid AI architecture, it’s essential to focus on metrics that reflect both latency and throughput across the inference pipeline.
- Time to First Token (TTFT) measures how quickly a system generates the first output token after receiving a prompt. This metric is primarily influenced by the prefill phase and directly impacts perceived responsiveness.
- Time Per Output Token (TPOT) reflects the time required to generate each subsequent token during the decode phase. Because decode is memory-bound, this metric is critical for understanding how efficiently the system handles continuous generation.
- Tokens Per Second (TPS) measures overall generation speed and throughput. It indicates how many tokens a system can produce per second, which is essential for supporting multiple concurrent agents at scale.
Together, these metrics provide a comprehensive view of performance in hybrid AI systems, helping organizations optimize both user experience and infrastructure efficiency.
How Does Hardware Influence Inference Speeds for Agents?
There are many steps that occur with handling an AI inference workload, but at the highest level, it can be broken into two primary steps: prefill and decode.
- The Prefill Phase (Compute-Bound): When a user or agent sends a prompt to a LLM, the input tokens are processed in parallel to build the Key-Value (KV) cache. Because the attention mechanism has a quadratic cost relative to input length, this phase is highly compute-intensive. The time it takes to compute the KV cache impacts the Time to First Token (TTFT), meaning the latency between the user hitting "enter" and the first character appearing.
- The Decode Phase (Memory-Bound): Once the prefill is complete, the model generates the output one token at a time (auto-regressively). Each new token requires reading the entire model weights and the growing KV cache from memory. Because the arithmetic intensity (math operations per byte of data loaded) is low, this phase is bottlenecked by memory bandwidth, not raw compute. The key metric here is how fast you can produce tokens, in other words, the Time Per Output Token (TPOT), and is measured in Tokens Per Second (TPS).
As our CTO Kunle Olukotun explained, for AI hardware, compute is the easy part. The challenging part, which Jensen also admitted, is optimizing memory movement – the most expensive part of running AI inference. SambaNova’s Dataflow Architecture is designed to overlay compute and data movement to deliver just-in-time to improve efficiency and performance.
For the compute-heavy portions of AI processing, GPUs are built to handle the highest amount of compute (the easy part) and are ideal for use cases like training and prefill. An example would be the Rubin CPX, which is purpose-built for prefill. However, decode is the current bottleneck for agents and is where alternative architectures are required.
The Future of Data Centers Is Hybrid Heterogeneous Hardware
For many AI use cases today, having a singular hardware stack for training AI models (GPUs) and a different hardware stack for inference (RDUs) makes the most sense.
However, for the use cases pushing the performance boundaries with agentic AI and the need to hit much higher performance requirements, inference providers must evaluate their AI clusters and determine how to segment them to deliver premium services to their customers. The future of data centers for these use cases will need to be segmented for the most common AI workloads: training, inference prefill, and inference decode.
Disaggregated Inference is an emerging tactic to be able to get the best of both GPUs and SambaNova’s RDUs. When a prompt is entered for an LLM, GPUs can turn the prompt into the fixed KV cache required for the decode phase.
This can be sent to a SambaRack to efficiently decode the output tokens with RDUs. For the end agentic system, this translates to lower time to first token, faster token output speed, and lower costs by batching many agentic requests together.
Why SambaNova RDUs Are the Best Choice for Inference and Decode
For agents to be effective, they need to be intelligent. Intelligence for agents comes from two factors: Large trillion-parameter models and faster token speeds to generate more thinking tokens. While GPUs can scale to large models, they are unable to hit the speed requirements of agents.
SRAM-only architectures, like Groq, suffer from capacity constraints, requiring thousands of chips with a complicated network configuration that is rigid and lacks the flexibility required for different AI workloads.
Due to our Dataflow Architecture and three-tier memory system, SambaNova RDUs are the only solution that is able to scale to large models, maintain fast decode tokens per second, and remain reconfigurable without a large infrastructure footprint.
Moreover, because of the system’s efficiency, it is able to fit in most existing data centers around the world, while delivering very competitive performance for agents. The three-tier architecture uniquely uses the right memory for each of these use cases.
- SRAM enables operator fusion to generate tokens.
- HBM enables cost-effective scale to store model weights on the KV cache that can be streamed to SRAM.
- DDR enables cost-effective prompt caching for agentic workloads to help reduce the TTFT for long-chained agents.
How to Get Started with RDUs
While the industry is just starting to talk about disaggregated inference with hybrid architectures, it is a complicated technique that only makes sense at sufficient scale and the most complicated agentic workflows. For inference providers who are looking to explore this capability, connect with our expert team on how we can collaborate together.
For those just looking to differentiate their infrastructure with faster inference,
SambaNova’s SN40L RDUs already provide fantastic performance for coding agents with models like MiniMax. You can try these on SambaCloud today in our Developer Tier.
For Inference service providers, connect with our team to see how you can integrate SambaStack with our existing inference engine.