One thing I'd add to the local inference discussion is that model weights aren't the whole memory budget. KV cache can become the actual bottleneck once you move from testing a model with one prompt to serving multiple users.
Take a self-hosted RAG service where several requests arrive with 32k-token contexts. Even if the quantized model weights fit comfortably in VRAM, concurrent sequences keep allocating KV cache as their contexts grow. Eventually, the scheduler has to preempt requests and recompute them, which can push tail latency up even though the model itself hasn't changed.
This is where vLLM's PagedAttention and prefix caching become interesting. They help manage KV memory and reuse shared prefixes, but they don't make the capacity problem disappear. I'd benchmark with realistic context lengths and concurrency, then watch KV cache utilization, preemption counts and p95 latency rather than just checking whether the model loads successfully.
A local model that fits isn't necessarily a local inference service that scales.
Custom software & AI development company. We build production-ready AI integrations, mobile apps, and web platforms for businesses that want real results. lycore.com
Great point about KV cache. We've seen the same thing: a model that loads fine in a single-prompt test can fall over once several long-context requests run concurrently. Tracking KV utilization, preemption counts and p95 latency under realistic concurrency is a much better readiness check than "it fits in VRAM", and we should have called that out in the post.
Exactly. I'd also look at how the scheduler handles mixed workloads, not just concurrent requests with similar context lengths.
For example, in a RAG service, one request might carry a 2k-token context while another needs 32k. If long-running sequences keep occupying KV blocks, shorter requests can end up waiting behind them even when their actual generation workload is relatively small.
I'd test different batching and scheduling policies under that kind of workload, especially chunked prefill and prefill/decode disaggregation. The interesting part is whether they improve time-to-first-token without pushing inter-token latency in the wrong direction.
Average throughput can look healthy while individual requests are having a pretty bad time.
Custom software & AI development company. We build production-ready AI integrations, mobile apps, and web platforms for businesses that want real results. lycore.com
That's a useful distinction: average throughput can hide head-of-line blocking when a 32k-token request sits next to a 2k one. We'd want to measure time-to-first-token per context-length bucket, so chunked prefill gets judged on the short requests it is meant to help. Splitting prefill and decode is on our test list for the same reason, with inter-token latency tracked alongside.
One thing I'd add is that chunk size itself deserves a closer look under mixed workloads. A smaller prefill chunk can reduce interference with active decode requests, but it also introduces more scheduling overhead and can hurt overall throughput if you push it too far.
For example, imagine a RAG workload where a 32k-token document ingestion request arrives during a burst of interactive queries. I'd test several max_num_batched_tokens values and compare TTFT and inter-token latency across context-length buckets, rather than assuming one chunk size works well for everything.
With prefill/decode disaggregation, I'd also track KV-cache transfer time between workers. Otherwise, you might eliminate GPU contention only to introduce a communication bottleneck elsewhere.
The metric I'd really want to see is goodput: how many requests actually meet their latency SLOs under sustained mixed traffic, not just how many tokens the system generates.
Custom software & AI development company. We build production-ready AI integrations, mobile apps, and web platforms for businesses that want real results. lycore.com
Good point on chunk size: it really is a trade-off curve, not a single best value. Sweeping max_num_batched_tokens and comparing TTFT and inter-token latency per context-length bucket is how we'd start too. Agreed on KV-cache transfer time with disaggregation as well; it's easy to move the bottleneck instead of removing it. Goodput against an explicit latency SLO is probably the honest headline metric, since raw tokens/sec hides the tail.
The real issue is latency. That's the true cost of local memory. Sounds counter intuitive, but 128gb of memory in a server, means more than 128gb of system memory at home, spread across 4 computers. It can be shared, sharded, repurposed, etc. Without ever being a problem to manage and it'll never go to waste. But that's only half the equation. What do you need memory for? To store game-states? To run a local LLM? Yet, you use it at most for 16 hours in a day actively and you will hardly ever actually utilize 100% of it. That's wastage. Especially when you consider in today's life, what you're actually using it for, is a local state clone of what's already running in the cloud? Whether it's viewing a website, local IDE state, a multiplayer game, etc. It's a waste of potential and often times, outright inefficient.
Take an online multiplayer game. CoD for instance, you need a pretty hefty pc to run it? What if it ran on the server and all your pc did was do a hash and render your game-state from that hash? You could run it at max detail on an iGPU, with next to no memory usage, if it were done properly. The cloud can run the actual game, with fragmented client states all in memory on the server. What that would mean, there's no 'lag' for inputs, only render latency. The server could with 100% certainty say who clicked first. No race conditions at all. It'd take a fraction of the compute and the only drawdown would be slightly higher network usage.
So yes, it's a waste having system memory, it wastes potential and it's built that way, because of an inefficient system.
Custom software & AI development company. We build production-ready AI integrations, mobile apps, and web platforms for businesses that want real results. lycore.com
Thanks for the thoughtful take. We agree that pooled, shared memory is usually used more efficiently than idle RAM spread across several home machines. Our "half right" view is that latency, privacy and offline work still push some client workloads local, so for us it comes down to what each workload actually needs. Streaming a rendered game state is an interesting idea, and the trade-off would be round-trip latency and network dependence.
True, but how much of that is due to existing paradigms and lack of innovation? Windows demands a minimum of 8gb of ram to run smoothly, but that's due to the local processing most things rely on, yet majority of people only use a computer for things they do online (or could do online). With the round-trip latency it's actually a catch-22, because currently, the same latency exists, except you get inconsistent behavior, because what your screen shows vs what the server ran, are off by eg. 200ms, that's probably the nr 1 reason people scream at their screens playing CSGO, because 64 tick servers are iffy at best determining who shot first and due to the cross-device latency, when you look at a replay with debug on, you'll see that what was on-target on your screen, was registered as off-target on the server. This would solve that permanently. Yes, network dependency would become a thing, but arent we already network dependent? Except with this system, you'd be able to run the latest AAA game on a potato pc, which opens up a much broader market and makes it more accessible. Same goes for CAD, no more need to have 64gb of ram to run a model, you run in cloud and it auto-scales?
There will always be tradeoffs, but imo the benefit far outweigh the drawdowns, take privacy and latency for instance, with a simple example, you're running an AI agent, in cloud, in a wasm process, that would be completely sandboxed and disposed of on completion, with near 0 latency execution time due to server proximity, your IDE would just render a canvas, as opposed to actually running anything. That speeds up your loop and you get more work done, equally as privately.
The only scenario where local memory is actually necessary, is offline work, which is a rare occurrence in 2026?
Custom software & AI development company. We build production-ready AI integrations, mobile apps, and web platforms for businesses that want real results. lycore.com
Fair challenge. Part of it is certainly habit: a lot of what runs locally is just inertia, and for browser-first work a thin client plus a fast connection is often enough. Where we still see local memory win is when the round trip is the bottleneck, such as interactive tools, large local files or offline use. So we'd treat it as a per-workload call rather than a rule either way.
This is a fairer take than most of the reactions I've seen. Altman's arithmetic about utilization is hard to argue with, but you're right that it answers the wrong question for a lot of client work. The reason I run something locally is almost never to save money on memory — it's because the data can't leave, or because I don't want a hosted model changing its policy, price or availability under a live product.
Your routing table is the useful part for me. Treating it as a routing question, and keeping the choice reversible in code, beats picking a side and defending it. To answer your closing question: anything with client financials, contracts under a strict NDA, or internal data I can't un-share would never go to a hosted API. For everything else, I start hosted for speed and move it only when the volume or the sensitivity earns it.
Custom software & AI development company. We build production-ready AI integrations, mobile apps, and web platforms for businesses that want real results. lycore.com
Thanks for sharing how you draw that boundary—client financials and NDA-bound material are exactly the cases where control matters more than raw token cost. Starting hosted for low-sensitivity work and moving only when volume or sensitivity justifies it is a practical way to keep the architecture reversible; we also like logging per-task cost and latency so that switch is evidence-based.
We're a place where coders share, stay up-to-date and grow their careers.
One thing I'd add to the local inference discussion is that model weights aren't the whole memory budget. KV cache can become the actual bottleneck once you move from testing a model with one prompt to serving multiple users.
Take a self-hosted RAG service where several requests arrive with 32k-token contexts. Even if the quantized model weights fit comfortably in VRAM, concurrent sequences keep allocating KV cache as their contexts grow. Eventually, the scheduler has to preempt requests and recompute them, which can push tail latency up even though the model itself hasn't changed.
This is where vLLM's PagedAttention and prefix caching become interesting. They help manage KV memory and reuse shared prefixes, but they don't make the capacity problem disappear. I'd benchmark with realistic context lengths and concurrency, then watch KV cache utilization, preemption counts and p95 latency rather than just checking whether the model loads successfully.
A local model that fits isn't necessarily a local inference service that scales.
Great point about KV cache. We've seen the same thing: a model that loads fine in a single-prompt test can fall over once several long-context requests run concurrently. Tracking KV utilization, preemption counts and p95 latency under realistic concurrency is a much better readiness check than "it fits in VRAM", and we should have called that out in the post.
Exactly. I'd also look at how the scheduler handles mixed workloads, not just concurrent requests with similar context lengths.
For example, in a RAG service, one request might carry a 2k-token context while another needs 32k. If long-running sequences keep occupying KV blocks, shorter requests can end up waiting behind them even when their actual generation workload is relatively small.
I'd test different batching and scheduling policies under that kind of workload, especially chunked prefill and prefill/decode disaggregation. The interesting part is whether they improve time-to-first-token without pushing inter-token latency in the wrong direction.
Average throughput can look healthy while individual requests are having a pretty bad time.
That's a useful distinction: average throughput can hide head-of-line blocking when a 32k-token request sits next to a 2k one. We'd want to measure time-to-first-token per context-length bucket, so chunked prefill gets judged on the short requests it is meant to help. Splitting prefill and decode is on our test list for the same reason, with inter-token latency tracked alongside.
One thing I'd add is that chunk size itself deserves a closer look under mixed workloads. A smaller prefill chunk can reduce interference with active decode requests, but it also introduces more scheduling overhead and can hurt overall throughput if you push it too far.
For example, imagine a RAG workload where a 32k-token document ingestion request arrives during a burst of interactive queries. I'd test several
max_num_batched_tokensvalues and compare TTFT and inter-token latency across context-length buckets, rather than assuming one chunk size works well for everything.With prefill/decode disaggregation, I'd also track KV-cache transfer time between workers. Otherwise, you might eliminate GPU contention only to introduce a communication bottleneck elsewhere.
The metric I'd really want to see is goodput: how many requests actually meet their latency SLOs under sustained mixed traffic, not just how many tokens the system generates.
Good point on chunk size: it really is a trade-off curve, not a single best value. Sweeping max_num_batched_tokens and comparing TTFT and inter-token latency per context-length bucket is how we'd start too. Agreed on KV-cache transfer time with disaggregation as well; it's easy to move the bottleneck instead of removing it. Goodput against an explicit latency SLO is probably the honest headline metric, since raw tokens/sec hides the tail.
The real issue is latency. That's the true cost of local memory. Sounds counter intuitive, but 128gb of memory in a server, means more than 128gb of system memory at home, spread across 4 computers. It can be shared, sharded, repurposed, etc. Without ever being a problem to manage and it'll never go to waste. But that's only half the equation. What do you need memory for? To store game-states? To run a local LLM? Yet, you use it at most for 16 hours in a day actively and you will hardly ever actually utilize 100% of it. That's wastage. Especially when you consider in today's life, what you're actually using it for, is a local state clone of what's already running in the cloud? Whether it's viewing a website, local IDE state, a multiplayer game, etc. It's a waste of potential and often times, outright inefficient.
Take an online multiplayer game. CoD for instance, you need a pretty hefty pc to run it? What if it ran on the server and all your pc did was do a hash and render your game-state from that hash? You could run it at max detail on an iGPU, with next to no memory usage, if it were done properly. The cloud can run the actual game, with fragmented client states all in memory on the server. What that would mean, there's no 'lag' for inputs, only render latency. The server could with 100% certainty say who clicked first. No race conditions at all. It'd take a fraction of the compute and the only drawdown would be slightly higher network usage.
So yes, it's a waste having system memory, it wastes potential and it's built that way, because of an inefficient system.
Thanks for the thoughtful take. We agree that pooled, shared memory is usually used more efficiently than idle RAM spread across several home machines. Our "half right" view is that latency, privacy and offline work still push some client workloads local, so for us it comes down to what each workload actually needs. Streaming a rendered game state is an interesting idea, and the trade-off would be round-trip latency and network dependence.
True, but how much of that is due to existing paradigms and lack of innovation? Windows demands a minimum of 8gb of ram to run smoothly, but that's due to the local processing most things rely on, yet majority of people only use a computer for things they do online (or could do online). With the round-trip latency it's actually a catch-22, because currently, the same latency exists, except you get inconsistent behavior, because what your screen shows vs what the server ran, are off by eg. 200ms, that's probably the nr 1 reason people scream at their screens playing CSGO, because 64 tick servers are iffy at best determining who shot first and due to the cross-device latency, when you look at a replay with debug on, you'll see that what was on-target on your screen, was registered as off-target on the server. This would solve that permanently. Yes, network dependency would become a thing, but arent we already network dependent? Except with this system, you'd be able to run the latest AAA game on a potato pc, which opens up a much broader market and makes it more accessible. Same goes for CAD, no more need to have 64gb of ram to run a model, you run in cloud and it auto-scales?
There will always be tradeoffs, but imo the benefit far outweigh the drawdowns, take privacy and latency for instance, with a simple example, you're running an AI agent, in cloud, in a wasm process, that would be completely sandboxed and disposed of on completion, with near 0 latency execution time due to server proximity, your IDE would just render a canvas, as opposed to actually running anything. That speeds up your loop and you get more work done, equally as privately.
The only scenario where local memory is actually necessary, is offline work, which is a rare occurrence in 2026?
Fair challenge. Part of it is certainly habit: a lot of what runs locally is just inertia, and for browser-first work a thin client plus a fast connection is often enough. Where we still see local memory win is when the round trip is the bottleneck, such as interactive tools, large local files or offline use. So we'd treat it as a per-workload call rather than a rule either way.
This is a fairer take than most of the reactions I've seen. Altman's arithmetic about utilization is hard to argue with, but you're right that it answers the wrong question for a lot of client work. The reason I run something locally is almost never to save money on memory — it's because the data can't leave, or because I don't want a hosted model changing its policy, price or availability under a live product.
Your routing table is the useful part for me. Treating it as a routing question, and keeping the choice reversible in code, beats picking a side and defending it. To answer your closing question: anything with client financials, contracts under a strict NDA, or internal data I can't un-share would never go to a hosted API. For everything else, I start hosted for speed and move it only when the volume or the sensitivity earns it.
Thanks for sharing how you draw that boundary—client financials and NDA-bound material are exactly the cases where control matters more than raw token cost. Starting hosted for low-sensitivity work and moving only when volume or sensitivity justifies it is a practical way to keep the architecture reversible; we also like logging per-task cost and latency so that switch is evidence-based.