Inference buyers increasingly care about ROI. That shift favors efficiency, repurposing existing data centers, and a hybrid world of chips, models, and partners.
Watch Rodrigo Liang’s conversation with Bloomberg’s Dani Burger at the CAIS Alternative Investment Summit for more on inference economics and what comes next.
Rodrigo Liang spoke with Dani Burger on Bloomberg Tech, from the CAIS Alternative Investment Summit in Beverly Hills, about AI inference, CapEx and what’s next for AI infrastructure. The conversation centered on how AI data centers will need to evolve to meet the ever-increasing demand for inference. Here are four key takeaways that every investor and operator in this market should hear:
In inference, economics is paramount. As you scale, it comes down to how quickly you can deliver returns on the capital you’re investing. Our SN50 payback target is six months, assuming 70% billable utilization. That comes from generating more tokens, for more users at the same time, on the same rack. As the market grows, people are going to become much more conscious of dollars and of return on investment.
Despite all this buildout, we’re still in the early stages. Most enterprises are just getting started delivering AI applications. Now we’re entering a phase defined by efficiency. At SambaNova, we focus on helping companies make better use of existing data centers. We want to deliver technology that scales, and that people can use without substantial extra cost.
Efficiency applies to capital, too. This is a capital-intensive business. As you scale, you’re putting chips into production and waiting nine to 12 months to stand up data centers, and all of that is capital still waiting to deliver value. We need to accelerate the time to value to meet the growing demands of AI-hungry applications.
One way to do that is to build where the infrastructure already exists. We’re partnering with Vista and Cambium Capital on Vector Core Compute (VC2), the inference cloud they launched in June. SambaNova powers it, and the strategy is to reuse and re-energize existing data centers. These brownfield sites already have the space, the power, and the infrastructure, and they exist all over the world. We just have to fit into that power footprint. Our air-cooled SambaRack systems draw 10 kilowatts (kW) per rack instead of 100 kW, and they are designed for exactly this infrastructure of existing data center facilities.
Reusing existing facilities can also reduce the need for new construction. We all have to be more conscious about how we scale AI data centers, and we’re doing our part: driving the power down, driving the cost down, and updating data centers that already exist.
None of this means inference will run on a single chip type. As inference grows, people are looking at heterogeneous environments, mixing chips from NVIDIA, AMD, and SambaNova to drive the best economics. The AI data centers coming online will likely be hybrid, and we’re excited to be part of that. Our disaggregated inference demo shows how hybrid inference works: NVIDIA GPUs process the prompt, and SambaNova’s Reconfigurable Dataflow Units (RDUs) runs the decode, and CPUs orchestrate processing.
The same is true of models. The frontier models are phenomenal pieces of technology, and we work closely with the model makers building them. But there are things companies want to keep private and secure, so they create their own proprietary models and run them on SambaNova, on-premises.
The same hybrid approach applies to our SambaNova partnerships. We’re partnering with Intel and several others, because you need more than one piece of technology. Creating the next level of AI inference ecosystem requires the best combination of RDUs, GPUs, and CPUs to ensure users receive the optimal speed and output for their AI programs.