Updated 2026-11-03: expanded the setup for all three runners (install, concrete models, a working server) and added a Troubleshooting section, based on reader feedback.
Running a model locally sounds like a weekend project. The setup is actually ten minutes, plus however long the model download takes on your connection. This is the copy-paste version: install the tool, pull a model, serve it over an OpenAI-compatible endpoint, and make your first tool call, with no hand-waving.
Treat this as a starting point for getting a model running, not a fine-tuning guide. The aim is a working endpoint in ten minutes; the deeper performance knobs (sampler settings, speculative decoding, GPU offload splits) are their own rabbit hole, and a separate post.
Why local, when hosted models are right there and genuinely good? Privacy is the obvious one: nothing leaves the machine. Then cost, because a long or repetitive job stops metering you by the token. And control, which matters more than it sounds. No rate limits, no refusal out of nowhere, and the version stays pinned instead of shifting under you one morning. You pay for all of it in quality and speed, and that price is real. The last section is about when it's worth paying.
First decision: which runner
Three tools cover almost everyone. Pick by how you work:
| Tool | Best for | Interface |
|---|---|---|
| Ollama | simplest CLI, scripting |
ollama run, HTTP API |
| LM Studio | GUI + a great CLI (lms), easy model management |
app + lms + OpenAI API |
| llama.cpp | maximum control, headless servers |
llama-server, raw flags |
I'll use LM Studio's CLI (lms) as the main path, because it gives you a clean OpenAI-compatible server and stays out of your way on model management, then show the Ollama and llama.cpp equivalents.
Sizing: will it even fit?
Before pulling anything, do the memory math. A quantized model's file size is roughly its RAM/VRAM footprint, plus headroom for the context (KV cache).
Rule of thumb: model file size + ~20% ≤ your available RAM/VRAM.
- 8 GB → a 7-8B model at Q4 (≈4-5 GB).
- 16 GB → a 13-14B at Q4, or an 8B comfortably.
- 32 GB → a ~27-34B at Q4.
- 64 GB → a 70B at Q4 starts to be realistic.
Quantization (Q4/Q5/Q8) trades quality for size. Q4_K_M is the usual sweet spot; go Q8 only if you have the memory to spare and want the extra fidelity.
LM Studio (via the lms CLI)
Install the LM Studio app from lmstudio.ai, launch it once, then put the bundled lms CLI on your PATH:
# macOS / Linux: bootstrap the CLI that ships with the app
~/.lmstudio/bin/lms bootstrap
# Windows (PowerShell), from that same bin folder: lms.exe bootstrap
# open a NEW terminal, then confirm it's on PATH
lms --version
Now download a concrete model, load it, and start the server:
# Download a model. Most reliable: the LM Studio app's search/download tab, or
# import a GGUF you grabbed from Hugging Face. The `lms get` CLI also works, but
# is flaky on some builds:
lms get llama-3.1-8b --yes # or pin a quant: lms get llama-3.1-8b@q4_k_m
# multi-GB download; can take well over ten minutes on a slow link (the model, not the setup).
# If `lms get` errors out (known on some versions), download the GGUF from Hugging
# Face and import it in the app, then continue below.
# load it, with an explicit context length for headroom
lms load llama-3.1-8b --context-length 16384 --yes
# start the server, pinned to a known port (the default can land on 1235, etc.)
lms server start --port 1234
lms server status # confirms the actual port and the loaded model
Housekeeping, when you need it:
lms ls # models on disk
lms ps # what's loaded right now
lms unload llama-3.1-8b # free the memory
lms server stop # stop the API server
With the server running, LM Studio serves an OpenAI-compatible API at http://localhost:1234/v1 (if you didn't pin --port, use whatever port lms server status reports; it is not always 1234). That compatibility is the whole point: anything that speaks the OpenAI API (your scripts, LiteLLM, an agent framework) points at the endpoint with a dummy key and works unchanged.
# confirm the server and see loaded models (needs jq: brew install jq / apt install jq)
curl -s http://localhost:1234/v1/models | jq '.data[].id'
Use whatever id that prints as the model value below, and make sure a model is actually loaded (lms ps). A first chat completion:
curl -s http://localhost:1234/v1/chat/completions \
-H "Authorization: Bearer lm-studio" \
-H "Content-Type: application/json" \
-d '{
"model": "llama-3.1-8b",
"messages": [{"role":"user","content":"Say hi in five words."}]
}' | jq -r '.choices[0].message.content'
The same thing in Ollama
Install Ollama, then pull a model and start the server:
# install: macOS `brew install ollama` (or the app); Linux:
curl -fsSL https://ollama.com/install.sh | sh
# download a model (same multi-GB caveat as above)
ollama pull llama3.1:8b
# start the API server on :11434 (the desktop app also runs it)
ollama serve
Ollama exposes an OpenAI-compatible endpoint at http://localhost:11434/v1, so the same curl calls as above work with the port swapped and the model set to llama3.1:8b (Ollama's tag, not the LM Studio key). If you just want to poke at the model, ollama run llama3.1:8b drops you straight into an interactive chat.
The same thing in llama.cpp
For headless servers or maximum control. Install it (macOS brew install llama.cpp; other platforms build from the repo), then let llama-server pull a real GGUF straight from Hugging Face and serve it:
# auto-download + serve a concrete model (the -hf flag fetches from Hugging Face)
llama-server -hf bartowski/Meta-Llama-3.1-8B-Instruct-GGUF:Q4_K_M \
-c 16384 --host 127.0.0.1 --port 8080 # localhost only; use 0.0.0.0 only to expose on your LAN
# prefer to manage the file yourself? download it, then point -m at the path:
# huggingface-cli download bartowski/Meta-Llama-3.1-8B-Instruct-GGUF \
# Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf --local-dir ./models
# llama-server -m ./models/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf -c 16384 --port 8080
llama-server also speaks the OpenAI API (/v1/chat/completions), on port 8080 here. One thing to watch on long sessions: the KV cache grows with context and RSS climbs over time; if latency degrades, restart the server to reset it.
Tool calling (the part most local models fumble)
If you want the model to do things (call functions, drive tools), you need tool-calling, and local models vary wildly here. Send the OpenAI tools array and see if it responds with a structured tool_calls (the call below targets LM Studio on :1234; swap the port and model for Ollama or llama.cpp):
curl -s http://localhost:1234/v1/chat/completions \
-H "Authorization: Bearer lm-studio" -H "Content-Type: application/json" \
-d '{
"model": "llama-3.1-8b",
"messages": [{"role":"user","content":"What time is it in UTC?"}],
"tools": [{
"type":"function",
"function":{
"name":"get_time",
"description":"Return the current time for a timezone",
"parameters":{"type":"object","properties":{"tz":{"type":"string"}}}
}
}],
"tool_choice":"auto"
}' | jq '.choices[0].message.tool_calls'
If you get a clean tool_calls array back, the model can drive tools. If you get prose describing the call instead, that model (or that quant) isn't reliable for agentic work; pick another, or keep the tool schema small and simple. This is the single biggest gotcha with local models, and it's worth its own article.
Choosing honestly
Local is not the default answer. Here's how I'd actually decide:
- Go local when the data cannot leave the machine, when the job is high-volume or repetitive, or when you need a pinned model that nothing rate-limits.
- Stay hosted when you need top-tier reasoning, reliability across long chains of steps, or you simply don't want to babysit memory and quantization.
- Most real setups land in between: local for the bulk, a hosted model for the handful of genuinely hard steps.
Troubleshooting
A few things that bite people (several surfaced from readers, thank you):
-
lms getfails to download. The CLI downloader is flaky on some LM Studio builds. Download the model in the LM Studio app's search/download tab instead, or import a GGUF you pulled from Hugging Face, thenlms loadit. -
The server isn't on
:1234. LM Studio falls back to the next free port (often:1235). Pin it withlms server start --port 1234, or runlms server statusto see the real port, then use that port in every request. -
/v1/modelsworks but/v1/chat/completionserrors. Almost always one of: no model is loaded (lms load …, confirm withlms ps); themodelfield doesn't match an id exactly as/v1/modelsprints it; or you're on the wrong port. -
Ollama:
ollama servesays the address is already in use. The desktop app already runs the server on:11434. Skipollama serve, or quit the app first. -
curlerrors withcommand not found: jq. Installjq(brew install jq/apt install jq); the snippets pipe through it. -
Tool-calling returns prose instead of
tool_calls. That model or quant can't drive tools reliably. Pick another, or keep the tool schema small and simple. -
llama.cpp slows down over a long session. The KV cache grows and RSS climbs; restart
llama-serverto reset it. -
Out of memory, or it crashes on load. The model plus its KV cache overran your RAM/VRAM. Drop to a smaller quant (Q4_K_M), lower
--context-length, or pick a smaller model (see the sizing table above). - Generation is painfully slow. Usually the model spilled out of RAM/VRAM to CPU or swap. Make it fit (smaller quant or model, lower context) and close other memory-hungry apps. A long first-token delay on big prompts is normal.
- The model downloaded but won't load. Re-download in case it was truncated, confirm it's a GGUF (for llama.cpp), and update LM Studio and its runtime so the format is supported.
-
Calling the endpoint from a browser app fails (CORS). Local servers usually don't send CORS headers. Call it from your backend or a small proxy, not straight from front-end
fetch(). -
401/ auth errors. These servers accept any dummy key. If a client insists on one, pass any non-empty string (lm-studio,ollama, whatever). -
A Hugging Face model is gated (download or
-hffails with 401/403). Plenty of models are free but still require you to accept a usage agreement before you can download them. Being logged in is not enough; you have to click through the form: - Open the model's page on huggingface.co while signed in and accept the license / usage agreement shown at the top of the page. Access is usually granted instantly.
- Create a token at huggingface.co → Settings → Access Tokens (a Read token is enough). Install the CLI if needed:
pip install -U "huggingface_hub[cli]". - Log in locally:
huggingface-cli login(paste the token when prompted), or on newer versionshf auth login. As an alternative, exportHF_TOKEN=<token>. After that,huggingface-cli download …andllama-server -hf …work for that repo. The community GGUF repos used above are ungated, so you can skip all of this for them. -
Ollama: the API can't find the model. The
modelfield must be the exact Ollama tag (llama3.1:8b), and you mustollama pullit first.
Takeaway
Pick a runner, do the memory math, load a model, hit the endpoint. That part really is ten minutes. The decisions that matter come after: how far you quantize, and whether the model can hold tool-calling, because that is what makes or breaks anything agentic you put on top of it.
Next: why local models fail at tool-calling, and how to design around it.






Top comments (0)