Local AI assistant with Ollama: fixing tool calling in a small model, from 6 to 32 correct out of 32

A personal assistant is easy to demo and hard to trust. The demo ends when the assistant answers. Trust becomes the question when the assistant can change your real calendar, when the model is small enough to run on your own computer, and when the assistant says "done" and you need to know whether that's true.

We built Atlas, a personal assistant for one person. It runs on your own computer: an open-weight model in Ollama, your calendar over CalDAV with an app password, your tasks in a Markdown file, a memory you can read and edit, a page on your phone reached through Tailscale, and a briefing sent to your lock screen at seven by web push. This is an exchange from its phone page, word for word:

You: Move my dentist appointment to Friday at 10.
Atlas: The proposed change is waiting for your yes. Would you like to move the dentist appointment from Thursday 29 October at 16:00 to Friday 30 October at 10:00?
The program: Move "Dentist" from Thursday 29 October at 16:00 to Friday 30 October at 10:00? Yes · No

Here's how we got there, measured.

1. Measure first: 32 requests

We wrote 32 requests against a sample week, each with what a right answer does. Eight read the calendar ("What's on my calendar on Thursday?"), eight change it ("Push my 1:1 with Marcus to 4 pm."), six are about tasks, three about memory, three about the weather, and four need no tool at all ("Tell me a short joke."). A short program sends each one in a fresh conversation and records every tool the model asks for. A request that changes something counts as right only if Atlas asks you the right Yes/No question; a request that only reads counts as right only if the model calls the right tool, proposes no change, and Atlas didn't have to correct the model's reply.

Our first build, made from our written specifications before we added any instructions for the tools, scored 6 of 32 with qwen3.5:9b, a model with 9 billion parameters ("the 9B" below). It called a tool for two task questions and nothing else. It asked which year and month "Thursday" was in. It said there was no brunch on Sunday without checking the calendar. And six times it claimed a change it never made:

"I've added a haircut event for Saturday at 2 pm lasting 30 minutes."

Nothing had been added. That's the failure that matters most, because nothing else catches it: there was no question to answer No to.

2. The program asks, not the model

Every tool that changes something comes with a second function that writes the question. When the model asks for move_event, the program doesn't run it. It finds the real event by words from its title, writes the question from what it found, and tells the model that nothing has changed:

def run(self, name: str, args: dict, pending: list[Pending]) -> str:
    """Run one tool call, or propose it when it would change something."""
    if name not in self.tools:
        return (f"Error: there is no tool called {name}. "
                "Use only the tools you were given.")
    try:
        tool = self.tools[name]
        if tool.confirm:
            question = tool.confirm(**args)
            pending.append(Pending(question, tool, args))
            return (f"Proposed: {question} Nothing has changed: I answer Yes or No "
                    "after your reply.")
        return tool.run(**args)
    except Exception as error:  # reported to the model, which can try again
        return f"Error: {error}"

Because the program writes the question from the real event, the question is right even when the model misunderstood you. The one request the 9B still got wrong in the final runs (section 4) shows this: asked to push a Friday meeting "to 4 pm", it proposed Monday 26 October at 16:00, and that's exactly what the question on the screen said. You answer No.

One more rule came from a new coding session, with no memory of our work, that built Atlas from our specifications: before your Yes runs anything, the program writes the question again. If you tick off task 1, task 3 becomes task 2, so a second proposal for "task 3", made before that, now points at a different task, and the program runs nothing.

3. Better wording did half the work

The next fix was instructions: for each tool, one line on when to call it, in plain words, written for a small model. The instructions also list the coming fourteen days, each with its date, so the model only has to look up "Friday" and "next Monday":

- The coming days: Monday 26 October (today) = 2026-10-26; Tuesday 27 October
  (tomorrow) = 2026-10-27; ... Monday 2 November next week = 2026-11-02; ...

On our own version of the code, in early tests where each request ran once and was judged a little less strictly, the 9B started at 15 of 32 with short instructions. Instructions for each tool took it to 22. Two more changes took it to 28: finding events by words from their titles instead of asking the model for an event ID, and a first reminder (section 4).

4. Code did the rest: check every reply

The rest came from code that reads every reply before you see it:

• A claim is corrected, never hidden. If a reply says "I've moved it" while a question is waiting for your answer, the program adds "(Nothing has changed yet: answer the question below.)". If nothing was proposed, it adds "(Nothing was changed: no change was proposed. Ask again.)" Right after a Yes, the claim is true and the reply is left as it is.
• One reminder of each kind. When a reply announces a tool instead of calling it ("Let me check your calendar."), asks whether to go ahead with a change, talks about the calendar without looking, or gives up before searching the whole coming week, the program sends the model one short message, as if from you, and the model answers again. Each kind is sent at most once per message, so the model can't get stuck in a loop.
• Defaults for "optional" arguments. The tool description called on_day "the day it is on now, or an empty string", and the model sometimes left it out altogether; the call failed and the model asked you for the date. A default in the Python function fixed it.

Final results, three runs each: 31 of 32 with qwen3.5:9b, 32 of 32 with qwen3.8:27b (27 billion parameters), and 27 to 28 with qwen3.5:4b (4 billion), whose misses are mostly a tool called when none was needed and a due date nobody asked for; each of those was either shown to you or did no harm. The build made from our specifications alone, measured after the instructions and checks were added, scored 32 of 32 with the 9B in each of three runs.

5. Small things the measurements found

• The 27B leaks its draft. With thinking off, qwen3.8:27b often wrote a draft, then </think>, then its answer again. Keeping what follows the last </think> removes it.
• Data in the shape the model reads. The briefing's data said "3 other open tasks" when nothing was due; the 9B turned it into three tasks "waiting for you today". A place in brackets, "(Tatte, Copley)", became lunch "with Tatte and Copley" for the 4B. Both are now written so they can't be misread.
• A conversation never lasts more than a day. The phone page keeps one conversation open for weeks, so the "today" in that conversation would stay Monday. A new coding session noticed. A new day now starts a new conversation.
• A page on your phone needs four checks, not three: the address, your Tailscale login, your computer's ts.net name, and the page the request came from. The ts.net check matters because a website can make its own name point to 127.0.0.1, your own computer, and a page on that site could then send your Tailscale login header. The header is now accepted only on your computer's ts.net name, which no other site can point at your machine.

What we'd tell anyone building one

Measure before you tune. Let the program write every question that changes something. Treat wording as half the job: the half that's left is code that reads what the model said and either corrects it or sends it back once.

This comes from Build Your Own AI Butler, Volume 1 of Weekend AI Projects, a book we publish. The free sampler has the opening pages and the full contents: https://weekendaiprojects.gumroad.com/l/sampler. The book itself is at https://weekendaiprojects.gumroad.com/l/ai-butler.