For the last few years, the AI conversation has been about capability: bigger models, longer context windows, better benchmarks. But if you’ve shipped anything with an LLM in production, you’ve already met the other constraint: every token costs something, and a lot of that cost is electricity.
In my last two posts, I looked at how energy markets ripple through cloud bills, and how AI is moving out of the data center and onto physical hardware. This post is where those threads meet. When power is expensive and compute is everywhere, efficiency stops being an optimization and becomes a design requirement.
Inference Is the Bill That Never Stops
Training gets the headlines, but inference is the workload that scales with your users. A model is trained once, then queried millions of times. Every feature you ship, from autocomplete to chat support to agentic workflows, adds to a recurring energy and compute bill.
Two things make this harder than traditional web workloads:
Requests aren’t uniform. A trivial lookup and a multi-step reasoning chain can hit the same endpoint but differ by orders of magnitude in compute.
Defaults favor waste. It’s easy to route everything to the biggest model because it “just works,” the same way we once over-provisioned servers because nobody was watching the bill.
The Efficiency Toolkit
The good news is that developers already have a growing set of levers.
Right-size the model. Not every task needs a frontier model. Classification, extraction, and routing are often handled well by smaller models, which are cheaper, faster, and far less power-hungry.
Route intelligently. A lightweight router that sends easy requests to small models and escalates only hard ones can cut spend dramatically without users noticing.
Quantize and distill. Reducing numerical precision and distilling larger models into smaller ones trades a modest accuracy hit for large gains in memory use, latency, and power draw.
Cache aggressively. Prompt caching and semantic caching mean you stop paying to recompute answers you’ve already produced.
Batch and stream wisely. Batching improves hardware utilization for non-interactive work. Streaming improves perceived latency for interactive work. Know which one you’re building.
Trim your tokens. Bloated system prompts and oversized context windows are the “unindexed database query” of the LLM era. Tokens you don’t send are tokens you don’t pay for.
Why This Connects to Physical AI
This is where the edge story matters. A robot or a phone can’t call out to an unlimited power supply. It has a battery, a thermal envelope, and a fixed NPU. Developers who learn to build within those limits now will be well placed as more AI moves on-device.
Edge deployment forces good habits that cloud abundance lets us skip:
Budget thinking: a fixed watt and memory budget instead of “scale it up.”
Latency discipline: local inference means no network round trip, but also no elastic capacity.
Graceful degradation: a smaller fallback model when power or heat is constrained.
Measure It or You’re Guessing
You can’t optimize what you don’t measure. A few practical starting points:
Track cost per successful task, not just cost per request.
Log tokens in and out per feature, and review the outliers.
Profile latency and throughput alongside quality metrics, since a cheaper model that fails twice as often isn’t cheaper.
Where tooling allows, estimate energy per request and watch the trend, even if the numbers are rough.
Efficiency without a quality check is just cutting corners. The goal is the best result per unit of compute.
Let’s Discuss
I’d like to hear from people running this in production:
Routing vs. one big model: Has model routing paid off for you, or did the added complexity eat the savings?
Measurement: What are you using to track cost or energy per request? Is there a tool you’d recommend?
Quality tradeoffs: Where have you found the line where a smaller or quantized model stops being good enough?
Culture: Does your team treat inference cost as an engineering metric, or is it still just “finance’s problem”?
Share your wins, failures, and hot takes in the comments. 👇
Sources
International Energy Agency, Energy and AI (executive summary): data center electricity demand and AI's role in its growth.
Anthropic, Prompt caching documentation: how prefix caching works and what it saves.
Zhang et al., EfficientLLM: Efficiency in Large Language Models: benchmarks of quantization and other compression methods.
Basta, Towards Green AI: Evaluating Energy Efficiency for Quantized LLM Inference (University of Waterloo, 2026): energy vs. latency vs. accuracy for quantized models.
Sustainable LLM Inference for Edge AI: quantized LLM energy and accuracy on edge hardware.
Dekoninck et al., A Unified Approach to Routing and Cascading for LLMs: routing queries across models of different sizes and costs.
SATER: A Self-Aware and Token-Efficient Approach to Routing and Cascading: routing and cascading strategies between small and large models.
Top comments (2)
Great post. The framing of inference as "the bill that never stops" is exactly right, and I think the comparison between bloated prompts and unindexed database queries will resonate with a lot of teams.
On your discussion questions, a few thoughts:
Routing vs. one big model: Routing pays off most when your traffic has a clear difficulty split, such as lots of classification or extraction mixed with a smaller tail of hard reasoning requests. Where it gets painful is the escalation logic. A router that misjudges difficulty creates a hidden quality cost, because the user gets a weaker answer and you never see the failure. Starting with simple rules (task type, input length) before moving to a learned router has been the safer path for many teams.
Measurement: Your point about tracking cost per successful task is the key one. Cost per request flatters cheap models that fail often. Pairing token logs with even a basic pass/fail eval per feature makes the tradeoffs much easier to see.
Culture: I'd add that inference cost tends to become an engineering metric only once it appears on the same dashboard as latency and error rate. While it lives in a finance report, nobody feels ownership of it.
One thing I'd love to see in a follow-up is a worked example showing before-and-after numbers for a single feature after applying caching, routing, and prompt trimming together. Seeing how much each lever contributes would help readers prioritize.
Thanks for connecting the cloud and edge threads so clearly. The graceful degradation point for on-device AI is an underrated design habit.
A brilliant perspective on a critical issue that deserves far more attention. As AI becomes increasingly integrated into our daily workflows, protecting privacy must go beyond restricting access to data; we must also consider what AI can uncover by connecting the dots. Security by design, not by instruction, is the key takeaway here. Excellent and thought-provoking read!