Stanford PhD students Ivanka and John present their groundbreaking research on "intelligence per watt" - a new metric for measuring AI energy efficiency. The researchers discuss their methodology for evaluating both local and cloud-based AI models, revealing surprising findings about the performance gap between small open-source models and frontier models being smaller than expected. Key discoveries include a 5.3x improvement in intelligence efficiency over two years and the potential for 80-90% resource savings through intelligent routing between local and cloud compute. The presentation covers their approach to measuring energy consumption during AI inference, the importance of intelligence per joule over watts, and introduces a router framework for optimizing workload allocation. The researchers emphasize the growing importance of energy efficiency in AI deployment as power constraints become a major bottleneck for scaling AI infrastructure, and discuss future directions including expanding benchmarks to cover agentic tasks and multi-turn conversations.
Efficiency in AI inference has mostly been measured in pieces. FLOPs per watt describes a chip. How many models you can host on a given number of chips describes a deployment. Neither connects the energy going in to the intelligence coming out. In this first ChipChats session, Vasanth Mohan of SambaNova sits down with Stanford researchers Avanika Narayan and Jon Saad-Falcon, authors of “Intelligence per Watt: Measuring Intelligence Efficiency of Local AI”, to unpack the metric they built to close that gap, what they found benchmarking model and hardware pairs, and what it means for anyone delivering intelligence inside a constrained power budget.
The metric itself is deliberately simple: intelligence per watt (IPW) is task accuracy divided by the power drawn to deliver it. One number for the end-to-end conversion of energy into useful intelligence.
Saad-Falcon describes the starting point as burgeoning inference demand. Projections from Google and NVIDIA suggested compounding year-over-year growth, and the strain that creates does not stop at silicon. It runs through networking infrastructure, auxiliary hardware, power and water sources, and the land required to build data centers at all.
Existing metrics measured pieces of that chain. FLOPs per watt describes a chip; models per node describes a deployment. Neither connects the energy going in to the intelligence coming out. Narayan is clear that the resulting picture is not a binary choice between cloud and local compute: demand is large enough that the answer is both, and IPW is how you measure the balance precisely enough to track it.
Intelligence is the easier half. The team selected benchmarks representative of the workloads that dominate real infrastructure today, single-turn chat and reasoning queries, then measured accuracy across a wide model set: closed-source models including GPT-5 and Claude Opus, alongside open-source families such as Qwen, IBM Granite and Llama.
Power is the harder half. Measurements come from each vendor’s own profiling stack, including NVML on NVIDIA hardware, AMD’s equivalent and SambaNova’s own profiler. Two inference settings were profiled so the results would be representative of both worlds: a batched, high-throughput configuration typical of data centers, using batches of roughly 32 or 64 queries in reduced precision, and a single-batch configuration typical of a local device serving one user. Both capture end-to-end latency and power draw from query load through prefill and decode to completion.
Watts measure power at an instant. Joules measure energy, which is power multiplied by time. Since publishing the paper, the team has moved toward the view that intelligence per joule is the more important of the two, because total energy volume is what determines whether infrastructure can actually scale.
It is also harder to measure, and Saad-Falcon is candid about why: instantaneous draw is a derivative, total energy is an integral. Power draw runs higher during prefill and lower during decode. There is idle draw while weights move in and out of memory. Auxiliary components such as networking and CPUs carry their own consumption. Harder, but more worthwhile, because total energy is what shows whether you can scale the infrastructure. Mohan draws out the practical consequence: if you can complete a task significantly faster and reduce the energy it consumes, that is what ultimately matters. SambaNova sets out the same arithmetic in Intelligence per Joule: the new metric for true AI value.
Narayan highlights three findings the team found genuinely surprising.
The hardware comparison is where the watt versus joule distinction becomes concrete. Measured on intelligence per watt, local and data center configurations were reasonably comparable, with local roughly half as efficient. Measured on intelligence per joule, the difference ran between 4× and 8×. Saad-Falcon points to the benchmarked SambaNova RDU infrastructure as a case where the contrast between the two metrics was especially dramatic: consumer GPUs from Apple and NVIDIA looked far worse on joules than on watts once set against purpose-built inference hardware.
One scope caveat the researchers state plainly: this work covers single-turn chat and reasoning. Extending it to multi-turn, tool-heavy agentic workloads produces more mixed results for local models, and that is where the research is heading next, alongside a finer-grained profiler and a follow-up metric they call gross domestic intelligence, which uses IPW to quantify how much compute power a nation holds.
Saad-Falcon describes a perfect router as the holy grail of inference workload management, a component that always knows which queries can be satisfied by local or on-premise compute and which genuinely need frontier capability. Projected across both single-turn and agentic workloads, the savings reach upwards of 80%, sometimes 90%, of the resources consumed. That is roughly a 10× improvement.
The encouraging part is that the router does not need to be perfect. Even routers around 60% to 70% accuracy capture much of the available saving, which happens to be where current state-of-the-art routers already operate, and production systems are shipping them: GPT-5 uses a router to delegate between models. Narayan grounds the argument in traffic composition. A large majority of queries hitting frontier services are not complex mathematics or scientific reasoning; they are writing, reformatting and similar tasks that smaller models handle well. Reformatting text in a particular tone does not require the most powerful model in the world.
Narayan’s guidance for enterprise teams is a change of posture. The default in recent years has been to send every query and every workload to a frontier model. Going forward, the discipline is intentional compute allocation, deciding deliberately which task runs where, for cost reasons and for power reasons.
Mohan adds the hardware dimension: there are many different architectures in the market, and they consume energy differently, some drawing more during prefill and some during decode. Narayan reads that heterogeneity as an advantage rather than a complication. We are fortunate to have chips specialized for different things, and chips that are fast, power efficient and optimized specifically for inference are the ones that should be running inference. That specialization across hardware providers is what makes a globally more efficient inference stack possible.
Mohan connects this to sovereign deployments, which may be power constrained in a specific geography. In those environments, getting the most intelligence out of every watt consumed matters more, not less. Both researchers also expect energy efficiency to climb the agenda generally: large data center build-outs are becoming visibly more energy conscious, hardware vendors are orienting their published metrics around efficiency, and on the model side the recent direction of travel has been toward token efficiency and production routers that delegate to smaller models.