Running Bigger AI Models, More Efficiently: The Real Production Challenge

TL;DR

• AI has moved from experimentation into production, and that shift exposes three hard constraints: models keep getting bigger, cost climbs steeply at scale, and power becomes a physical ceiling.
• On a recent Fortune panel, SambaNova CEO Rodrigo Liang and Adaption Labs CEO Sara Hooker agreed the industry's central problem is now efficiency, though they approach it from different angles: Liang from the infrastructure side, Hooker from the model-architecture side.
• Large models are not going away. For the most demanding workloads, they are unavoidable, so the real question is how to run them efficiently rather than whether to run them at all.
• Inference in the agentic era behaves nothing like the single-model throughput problem of the past. In Hooker's words, “inference is a different beast.” Constant data movement, not raw compute, is the bottleneck.
• SambaNova's focus is premium inference, defined on the panel by speed and size: Running the largest models much more efficiently, at full precision, rather than trading accuracy for speed.

Are Large Models Here to Stay?

The question hung over the whole conversation: Are today's largest foundation models where AI is heading, or will we look back on them as a detour? A recent Fortune panel titled “From the AI We Have to the AI We Need” put it directly to two people building very different answers, SambaNova co-founder and CEO Rodrigo Liang and Adaption Labs co-founder and CEO Sara Hooker, moderated by Fortune AI editor Jeremy Kahn.

They came at it from opposite ends of the stack. Hooker works on model architecture and the way models learn. Liang engineers the chips that run them. But they converged on the same diagnosis: The next phase of AI will be defined less by how large models can get and more by how efficiently we can run the large models we already depend on. Here is what the panel surfaced.

What the Panel Said: AI Has Entered Its Production Phase

Asked whether the industry is stuck with the largest models forever, Liang gave a both/and answer. He explained that the world is heterogeneous, so scale still matters, and large models are going to be here for a while, but there is still a need for a new generation of more efficient models. AI is entering its production phase, and production at scale runs into three constraints at once.

The Three Constraints of Production AI

First, the models keep getting bigger, at least the class of models that attract the most attention. Second, cost becomes a serious problem as you scale, and that cost is largely driven by the infrastructure available to run those models. Third, there is power. 

As Liang was careful to point out, the industry's scramble for chips and energy is not caused by small models. It is the large models that consume a disproportionate share of the compute, which is exactly why the pressure lands hardest at the top end.

The constraints do not point toward abandoning scale. They point toward running it far more efficiently..

The Model-Side View: Why the Workload Is Changing

Hooker approached the same problem from the model side. In her words, “Today's models are monolithic. They're stuck in time. A big goal of how we think about what should be fundamentally different is how do we efficiently learn from the environment, because now all models are moving towards interaction. Now it matters more that last mile performance, and for that you need models which can evolve, otherwise you end up with massive inefficiencies.”

She sees an inflection point arriving with real urgency. Part of it is not throwing the largest models at problems that do not need them. She emphasized, “You shouldn't just apply the same model to all problems. Probably 90% of problems are very easy. Many of the things that you do in bulk processing, for example, you shouldn't be throwing a massive model at. 10% you need a ton of compute.”

Her own focus, continuous learning, follows from this. As she defined it on the panel: “Continuous learning is really the goal that you should be able to update model behavior without forgetting what you already know.”

The two perspectives are complementary: efficiency as the key to improving AI. Hooker describes why the workload is changing. Liang describes what it takes to serve that workload.

Why Inference Has Become a Different Beast

For most of the last decade, AI infrastructure was tuned for training: one enormous model, one enormous workload, optimize for throughput and push data through. Hooker's point on the panel was that inference in the agentic era looks nothing like that. What changes everything is that agentic tasks force the system to keep returning to memory, she continues “You have to check in as you go. That is a very painful operation. It means it's basically the whole crisis with memory.”

The Oven Problem: Memory, Not Compute, Is the Bottleneck

She offered an analogy that stuck, “We can build the biggest oven in the world and our only bottleneck is how fast the baker can put stuff in the oven. The baker is memory. What's fascinating now is that we're basically taking things out and putting things in the oven multiple times over the course of the recipe.”

Her conclusion was blunt, “Inference is a different beast,” and the design choices around memory become the critical ones. Liang pushed the same analogy further: “I think we have 50 ovens and we're actually moving from one oven to another to another. And so, ideally, you want to find ways to keep things in the oven and not keep opening. It's like the way I actually do turkeys for Thanksgiving.”

The point underneath the metaphor: Every unnecessary transfer of data costs time and energy. This is the heart of why inference behaves the way it does at scale. The constant shuttling of data between compute and memory increases costs, and agentic AI multiplies that movement.

How SambaNova Approaches It: Premium Inference

If the panel diagnosed the problem, SambaNova's approach is one answer to it. Liang described the company's focus as premium inference, defined by two things: speed and size. Until natively efficient models arrive, he argued, the biggest models are the ones that have to run better, and its Reconfigurable Dataflow Unit (RDU) chip is built for exactly that. He added, “The battleground is in these very large trillion parameterized models and you've got to run those much much more efficiently than we are today.”

Run the Model as Its Creators Built It

One decision sets the approach apart, according to Liang. There are shortcuts to gain speed, such as quantizing a model down to lower precision, but as he noted, you lose accuracy that way. “The way we do it is we run everything original precision. If great researchers create a wonderful model, it's not for the hardware guys to come and just chop the accuracy. And so we run it full precision, and we run about 10 times faster at one-tenth the power.”

That result comes from the Dataflow Architecture inside the chip, which keeps intermediate results local and cuts the data movement the panel identified as the real cost. SambaNova also does not try to run everything. As Liang explained, “We don't necessarily come and say we do everything. What we do is we take the most popular models that people like and we optimize for that.”

Disaggregated Inference and Heterogeneous Compute

Liang also described the data centers SambaNova deploys into as heterogeneous: not only its own chips, but CPUs and even other companies' GPUs for different parts of the workload. He pointed to the first disaggregated inference demo conducted recently, where GPU-based hardware handles the prefill stage and SambaNova runs the decode of the same model. “On the largest models,” he added, “this delivers meaningful gains in both speed and power, which at scale is how the cost comes down. It is one example of matching each stage of a workload to the hardware that runs it best.”

The Right Model, on the Right Chip, for the Right Task

The panel concurred that there is no single silver bullet. Modern AI data centers are increasingly heterogeneous. Different workloads have different profiles, and a mature production stack routes each one to the hardware that runs it best rather than forcing everything through the same silicon. For enterprises weighing where to run production AI, SambaStack applies that principle across the full stack, from chip to model-serving API, with the largest models running inside a single air-cooled SambaRack.

Liang also made a point about flexibility that matters for anyone planning infrastructure. Hardware cycles run three to four years, so vendors ship today what they designed years ago, while researchers keep inventing things no one could have anticipated. “We're releasing things now that we created three years ago, and we have great minds like Sara who are inventing things that we could not possibly see three years earlier.”

His conclusion was to build general-purpose infrastructure with the right abstraction layer, so that architectures created in GPU-centric training ecosystems can move over and still get the benefit of the newer hardware. 

Hooker made the complementary case from the model side, on why inference is now where the work is: “You don't just want to scale anymore. Those are not the returns in the pre-training stage. Most labs but also most workloads are shifting towards how do you do things test time.” That shift toward test-time and interaction, she argued, is exactly why domain-specialized hardware is worth designing.

Efficiency Is the Bridge, and the Work Is Happening Now

Hooker closed the panel on a note about where the interesting problems now sit. “Progress,” she argued, “No longer comes from simply scaling. The formula now is not as simple as, like, just throw more weights at the problem. The era of research is back again.”

For model builders, she framed the shift as a change in what defines the work itself, “Your identity is not the weights. You're thinking about how fast and how efficiently can you get that model to learn from its environment.”

The trajectory the panel described is clear enough. Models will keep getting bigger. Agentic workloads will multiply the inference behind every application. And the constraint that decides which deployments reach production, cost and power at scale, is an efficiency problem. Running large models efficiently is the bridge that turns ambitious scale into something sustainable. You can read more on how SambaNova approaches this across its resources and research.

Want to see what full-precision, energy-efficient inference looks like on the largest models? Talk to an expert or explore the RDU chip behind SambaNova's approach.