TL;DR
Companies are bringing inference behind their own firewalls. What enables this now is technology that works with the data centers they already have.
Watch Rodrigo Liang’s conversation with Bloomberg’s Dani Burger at the CAIS Alternative Investment Summit for more on bringing inference in-house and why the future of AI is hybrid.
Rodrigo Liang spoke with Dani Burger on Bloomberg Tech, from the CAIS Alternative Investment Summit in Beverly Hills, about AI inference, CapEx and what’s next for AI infrastructure. Investors are regularly asking us a question: What happens to the big model providers if a company like JPMorganChase decides to train or fine-tune a model and run it behind the firewall?
It is a fair question, and in fact, that is exactly what is happening. We call it repatriation: infrastructure that went to the cloud coming back behind the company’s own gates. Our racks are running inference inside JPMorganChase’s firewall, keeping the bank’s customers’ data safe, and we expect this formula to extend well beyond banking.
For the model providers, we’d argue that the future will be hybrid. The frontier models are tremendous, and they are going to get even larger. We work closely with those labs ourselves. But companies want to keep certain pieces of data and information private and secure. They have their own IP, so they can use open-source or fine-tuned models and run them on-premises, with more control over where their data is processed.
Hybrid applies to the hardware, too, and SambaNova engineers with this in mind. Our RDUs (reconfigurable dataflow units) can work alongside GPUs, so companies can use different chips for different jobs.
Why companies are bringing inference home
To understand why companies are bringing inference behind their own firewalls, it’s worth understanding the difference between training and inference. Training is episodic: You assemble a data set one time, and then you can clean it, review it, and set parameters under which it operates. Inference is constant. Every time a question is asked, the model is reading sensitive information. If that process is running on someone else’s cloud, that information crosses firewalls countless times. The only way to protect this information is to keep inference running where the data already exists – behind a firewall.
Security is the first reason companies are bringing their data behind a firewall. The second is control. Companies do not want external resources to operate and control the infrastructure their business depends on. That kind of dependency can create business risk, and it’s partly why our approach has proven attractive. SambaNova builds the chips and racks, but they sit in the company’s building and behind its firewall. No outside provider can reprice, throttle, or shut them down.
Why on-premises inference is possible now
Wanting inference at home is one thing, but housing it is another. Building a new AI data center can take anywhere from nine to 12 months, and the power to run it may take even longer to build. But many companies already have data centers, with the controls for security, privacy, and compliance built, and that is where their data lives. The question is whether inference can fit inside them.
Usually, the answer is no. Many of those buildings were designed for racks that draw around 10 kilowatts (kW) and cool themselves with it. A typical AI rack draws 100 or more kilowatts and needs liquid cooling. It cannot go into the data center you already have, and so it waits for the one you have to build at great time and expense.
That’s why we decided to build power-efficient, air-cooled racks that can work in the data center you already have. It’s called SambaRack, and it runs on our RDUs, which cut down on data movement to help make inference more efficient.
Power is the constraint everywhere, not just in the enterprise. The frontier labs are hitting power caps too, where they simply cannot build dense data centers. Whoever can build the most powerful inference data center on the least power will solve it, for the labs and for the banks alike. We started there, and it’s what lets us bring inference back behind a firewall into data centers already owned by our customers, and protect their data where they've always protected it.