DEV Community

Cover image for Run a Capable LLM on Your Own Machine
Mikalai Karytka
Mikalai Karytka

Posted on Edited on

Run a Capable LLM on Your Own Machine

Updated 2026-11-03: expanded the setup for all three runners (install, concrete models, a working server) and added a Troubleshooting section, based on reader feedback.

Running a model locally sounds like a weekend project. The setup is actually ten minutes, plus however long the model download takes on your connection. This is the copy-paste version: install the tool, pull a model, serve it over an OpenAI-compatible endpoint, and make your first tool call, with no hand-waving.

Treat this as a starting point for getting a model running, not a fine-tuning guide. The aim is a working endpoint in ten minutes; the deeper performance knobs (sampler settings, speculative decoding, GPU offload splits) are their own rabbit hole, and a separate post.

Why local, when hosted models are right there and genuinely good? Privacy is the obvious one: nothing leaves the machine. Then cost, because a long or repetitive job stops metering you by the token. And control, which matters more than it sounds. No rate limits, no refusal out of nowhere, and the version stays pinned instead of shifting under you one morning. You pay for all of it in quality and speed, and that price is real. The last section is about when it's worth paying.

This is fine: quant, context length, KV cache, VRAM math. The ten-minute setup has a few knobs

First decision: which runner

Three tools cover almost everyone. Pick by how you work:

Tool Best for Interface
Ollama simplest CLI, scripting ollama run, HTTP API
LM Studio GUI + a great CLI (lms), easy model management app + lms + OpenAI API
llama.cpp maximum control, headless servers llama-server, raw flags

I'll use LM Studio's CLI (lms) as the main path, because it gives you a clean OpenAI-compatible server and stays out of your way on model management, then show the Ollama and llama.cpp equivalents.

One mental model: each runner serves an OpenAI-compatible endpoint on localhost that any OpenAI client can call

Sizing: will it even fit?

Before pulling anything, do the memory math. A quantized model's file size is roughly its RAM/VRAM footprint, plus headroom for the context (KV cache).

Rule of thumb: model file size + ~20% ≤ your available RAM/VRAM.

  • 8 GB → a 7-8B model at Q4 (≈4-5 GB).
  • 16 GB → a 13-14B at Q4, or an 8B comfortably.
  • 32 GB → a ~27-34B at Q4.
  • 64 GB → a 70B at Q4 starts to be realistic.

Quantization (Q4/Q5/Q8) trades quality for size. Q4_K_M is the usual sweet spot; go Q8 only if you have the memory to spare and want the extra fidelity.

Will it fit? Biggest comfortable model at Q4 per RAM tier: 8 GB fits 7-8B, 16 GB fits 13-14B, 32 GB fits 27-34B, 64 GB fits 70B

Buff Doge vs Cheems: 64 GB RAM runs a 70B at Q4, 8 GB RAM barely fits a 7B

LM Studio (via the lms CLI)

Install the LM Studio app from lmstudio.ai, launch it once, then put the bundled lms CLI on your PATH:

# macOS / Linux: bootstrap the CLI that ships with the app
~/.lmstudio/bin/lms bootstrap
# Windows (PowerShell), from that same bin folder: lms.exe bootstrap

# open a NEW terminal, then confirm it's on PATH
lms --version
Enter fullscreen mode Exit fullscreen mode

Now download a concrete model, load it, and start the server:

# Download a model. Most reliable: the LM Studio app's search/download tab, or
# import a GGUF you grabbed from Hugging Face. The `lms get` CLI also works, but
# is flaky on some builds:
lms get llama-3.1-8b --yes        # or pin a quant: lms get llama-3.1-8b@q4_k_m
# multi-GB download; can take well over ten minutes on a slow link (the model, not the setup).
# If `lms get` errors out (known on some versions), download the GGUF from Hugging
# Face and import it in the app, then continue below.

# load it, with an explicit context length for headroom
lms load llama-3.1-8b --context-length 16384 --yes

# start the server, pinned to a known port (the default can land on 1235, etc.)
lms server start --port 1234
lms server status                 # confirms the actual port and the loaded model
Enter fullscreen mode Exit fullscreen mode

Housekeeping, when you need it:

lms ls                      # models on disk
lms ps                      # what's loaded right now
lms unload llama-3.1-8b     # free the memory
lms server stop             # stop the API server
Enter fullscreen mode Exit fullscreen mode

With the server running, LM Studio serves an OpenAI-compatible API at http://localhost:1234/v1 (if you didn't pin --port, use whatever port lms server status reports; it is not always 1234). That compatibility is the whole point: anything that speaks the OpenAI API (your scripts, LiteLLM, an agent framework) points at the endpoint with a dummy key and works unchanged.

# confirm the server and see loaded models (needs jq: brew install jq / apt install jq)
curl -s http://localhost:1234/v1/models | jq '.data[].id'
Enter fullscreen mode Exit fullscreen mode

Use whatever id that prints as the model value below, and make sure a model is actually loaded (lms ps). A first chat completion:

curl -s http://localhost:1234/v1/chat/completions \
  -H "Authorization: Bearer lm-studio" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama-3.1-8b",
    "messages": [{"role":"user","content":"Say hi in five words."}]
  }' | jq -r '.choices[0].message.content'
Enter fullscreen mode Exit fullscreen mode

The same thing in Ollama

Install Ollama, then pull a model and start the server:

# install: macOS `brew install ollama` (or the app); Linux:
curl -fsSL https://ollama.com/install.sh | sh

# download a model (same multi-GB caveat as above)
ollama pull llama3.1:8b

# start the API server on :11434 (the desktop app also runs it)
ollama serve
Enter fullscreen mode Exit fullscreen mode

Ollama exposes an OpenAI-compatible endpoint at http://localhost:11434/v1, so the same curl calls as above work with the port swapped and the model set to llama3.1:8b (Ollama's tag, not the LM Studio key). If you just want to poke at the model, ollama run llama3.1:8b drops you straight into an interactive chat.

The same thing in llama.cpp

For headless servers or maximum control. Install it (macOS brew install llama.cpp; other platforms build from the repo), then let llama-server pull a real GGUF straight from Hugging Face and serve it:

# auto-download + serve a concrete model (the -hf flag fetches from Hugging Face)
llama-server -hf bartowski/Meta-Llama-3.1-8B-Instruct-GGUF:Q4_K_M \
  -c 16384 --host 127.0.0.1 --port 8080   # localhost only; use 0.0.0.0 only to expose on your LAN

# prefer to manage the file yourself? download it, then point -m at the path:
#   huggingface-cli download bartowski/Meta-Llama-3.1-8B-Instruct-GGUF \
#     Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf --local-dir ./models
#   llama-server -m ./models/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf -c 16384 --port 8080
Enter fullscreen mode Exit fullscreen mode

llama-server also speaks the OpenAI API (/v1/chat/completions), on port 8080 here. One thing to watch on long sessions: the KV cache grows with context and RSS climbs over time; if latency degrades, restart the server to reset it.

Tool calling (the part most local models fumble)

If you want the model to do things (call functions, drive tools), you need tool-calling, and local models vary wildly here. Send the OpenAI tools array and see if it responds with a structured tool_calls (the call below targets LM Studio on :1234; swap the port and model for Ollama or llama.cpp):

curl -s http://localhost:1234/v1/chat/completions \
  -H "Authorization: Bearer lm-studio" -H "Content-Type: application/json" \
  -d '{
    "model": "llama-3.1-8b",
    "messages": [{"role":"user","content":"What time is it in UTC?"}],
    "tools": [{
      "type":"function",
      "function":{
        "name":"get_time",
        "description":"Return the current time for a timezone",
        "parameters":{"type":"object","properties":{"tz":{"type":"string"}}}
      }
    }],
    "tool_choice":"auto"
  }' | jq '.choices[0].message.tool_calls'
Enter fullscreen mode Exit fullscreen mode

If you get a clean tool_calls array back, the model can drive tools. If you get prose describing the call instead, that model (or that quant) isn't reliable for agentic work; pick another, or keep the tool schema small and simple. This is the single biggest gotcha with local models, and it's worth its own article.

Anakin and Padme: my local model does tool calling, so it returns a real tool_calls array, right?

Choosing honestly

Local is not the default answer. Here's how I'd actually decide:

  • Go local when the data cannot leave the machine, when the job is high-volume or repetitive, or when you need a pinned model that nothing rate-limits.
  • Stay hosted when you need top-tier reasoning, reliability across long chains of steps, or you simply don't want to babysit memory and quantization.
  • Most real setups land in between: local for the bulk, a hosted model for the handful of genuinely hard steps.

Drake: no to a per-token bill and rate limits for a repetitive job, yes to local for the bulk and hosted for the hard parts

Troubleshooting

A few things that bite people (several surfaced from readers, thank you):

  • lms get fails to download. The CLI downloader is flaky on some LM Studio builds. Download the model in the LM Studio app's search/download tab instead, or import a GGUF you pulled from Hugging Face, then lms load it.
  • The server isn't on :1234. LM Studio falls back to the next free port (often :1235). Pin it with lms server start --port 1234, or run lms server status to see the real port, then use that port in every request.
  • /v1/models works but /v1/chat/completions errors. Almost always one of: no model is loaded (lms load …, confirm with lms ps); the model field doesn't match an id exactly as /v1/models prints it; or you're on the wrong port.
  • Ollama: ollama serve says the address is already in use. The desktop app already runs the server on :11434. Skip ollama serve, or quit the app first.
  • curl errors with command not found: jq. Install jq (brew install jq / apt install jq); the snippets pipe through it.
  • Tool-calling returns prose instead of tool_calls. That model or quant can't drive tools reliably. Pick another, or keep the tool schema small and simple.
  • llama.cpp slows down over a long session. The KV cache grows and RSS climbs; restart llama-server to reset it.
  • Out of memory, or it crashes on load. The model plus its KV cache overran your RAM/VRAM. Drop to a smaller quant (Q4_K_M), lower --context-length, or pick a smaller model (see the sizing table above).
  • Generation is painfully slow. Usually the model spilled out of RAM/VRAM to CPU or swap. Make it fit (smaller quant or model, lower context) and close other memory-hungry apps. A long first-token delay on big prompts is normal.
  • The model downloaded but won't load. Re-download in case it was truncated, confirm it's a GGUF (for llama.cpp), and update LM Studio and its runtime so the format is supported.
  • Calling the endpoint from a browser app fails (CORS). Local servers usually don't send CORS headers. Call it from your backend or a small proxy, not straight from front-end fetch().
  • 401 / auth errors. These servers accept any dummy key. If a client insists on one, pass any non-empty string (lm-studio, ollama, whatever).
  • A Hugging Face model is gated (download or -hf fails with 401/403). Plenty of models are free but still require you to accept a usage agreement before you can download them. Being logged in is not enough; you have to click through the form:
  • Open the model's page on huggingface.co while signed in and accept the license / usage agreement shown at the top of the page. Access is usually granted instantly.
  • Create a token at huggingface.co → Settings → Access Tokens (a Read token is enough). Install the CLI if needed: pip install -U "huggingface_hub[cli]".
  • Log in locally: huggingface-cli login (paste the token when prompted), or on newer versions hf auth login. As an alternative, export HF_TOKEN=<token>. After that, huggingface-cli download … and llama-server -hf … work for that repo. The community GGUF repos used above are ungated, so you can skip all of this for them.
  • Ollama: the API can't find the model. The model field must be the exact Ollama tag (llama3.1:8b), and you must ollama pull it first.

Takeaway

Pick a runner, do the memory math, load a model, hit the endpoint. That part really is ten minutes. The decisions that matter come after: how far you quantize, and whether the model can hold tool-calling, because that is what makes or breaks anything agentic you put on top of it.

Next: why local models fail at tool-calling, and how to design around it.

Top comments (0)