Best AI Bookkeeper: How to Score AI Categorization on Your Books

To score an AI that categorizes bank transactions and find the best AI bookkeeper for your books, label a sample of your own transactions with the category your accountant would use, then measure precision (of what the AI booked, how much was right) at several confidence thresholds, alongside coverage (how much it booked without asking). One "percent auto-categorized" figure hides the trade-off that matters: an AI that books everything is wrong more often than one that asks about the transactions it isn't sure of. The Python below computes both from a labelled sample.

Disclosure: I work on ZeraBooks, which does AI bookkeeping and ranks itself first in our comparisons. I'm not publishing our numbers here; the point is to measure any tool, including ours, on your own data.

Why is "98% auto-categorized" not an accuracy number?

Because it usually measures coverage, not correctness. "Auto-categorized" means the AI put a category on the transaction. It doesn't say the category was right.

Vendors publish figures like this on their pricing pages. Puzzle, for example, lists 85 to 90% of transactions categorized automatically on its $30 Starter plan and up to 98% across its range (checked November 8, 2026). That's a useful number for "how much will I have to touch", and it's a fine thing to publish. It's just a different question from "how much of what it touched was right", and you have to measure that one yourself.

Why your own measurement will differ from anyone's headline:

• Your chart of accounts. "Materials" vs "Cost of goods sold" vs "Job supplies" is a choice your accountant made.
• Your business. Home Depot is materials for a landscaper and might be an owner draw on a Saturday.
• Your history. A tool that learns from your corrections should score differently in month three than in month one.
• The hard rows are rare and expensive. Transfers, loan payments and checks with no payee are a small share of rows and a big share of what an accountant fixes at year end.

Build a labelled sample

Take a recent month or two. For each transaction, record the description, the AI's category, the AI's confidence if the tool exposes one, and the correct category as your accountant would book it. A few hundred rows is enough to see the shape. The sample below has sixteen fictional rows from a landscaping business, deliberately including the hard cases.

from collections import Counter

# (description, what the AI picked, its confidence, what the accountant says is right)
SAMPLE = [
    ("ADP PAYROLL 0915", "Payroll", 0.99, "Payroll"),
    ("ADP PAYROLL 0930", "Payroll", 0.99, "Payroll"),
    ("ADP PAYROLL FEES", "Payroll", 0.91, "Bank and payroll fees"),
    ("SITEONE LANDSCAPE", "Materials", 0.95, "Materials"),
    ("HOME DEPOT 4410", "Materials", 0.88, "Materials"),
    ("HOME DEPOT 4410", "Materials", 0.62, "Owner draw"),
    ("DUKE ENERGY", "Utilities", 0.97, "Utilities"),
    ("VERIZON WIRELESS", "Phone and internet", 0.96, "Phone and internet"),
    ("QT 1146 FUEL", "Fuel", 0.93, "Fuel"),
    ("JOHN DEERE FINANCIAL", "Equipment expense", 0.58, "Loan principal and interest"),
    ("TRANSFER FROM SAV 4410", "Sales", 0.41, "Owner contribution"),
    ("CHECK 3021", "Contractors", 0.35, "Contractors"),
    ("CHECK 3022", "Contractors", 0.33, "Rent"),
    ("STRIPE TRANSFER", "Sales", 0.98, "Sales"),
    ("STRIPE TRANSFER", "Sales", 0.98, "Sales"),
    ("MONTHLY SERVICE FEE", "Bank and payroll fees", 0.97, "Bank and payroll fees"),
]


def at_threshold(rows, t):
    """The AI books what it is at least t sure of and asks about the rest."""
    booked = [r for r in rows if r[2] >= t]
    right = sum(1 for _, guess, _, truth in booked if guess == truth)
    coverage = len(booked) / len(rows)
    precision = right / len(booked) if booked else 1.0
    return coverage, precision, len(rows) - len(booked), len(booked) - right


print("threshold  booked  precision  questions  booked but wrong")
for t in (0.0, 0.6, 0.8, 0.9, 0.95):
    cov, prec, asked, wrong = at_threshold(SAMPLE, t)
    print(f"{t:>9}  {cov:>6.0%}  {prec:>9.0%}  {asked:>9}  {wrong:>15}")

print("\nper category, everything booked (threshold 0):")
picked = Counter(g for _, g, _, _ in SAMPLE)
actual = Counter(t for _, _, _, t in SAMPLE)
hits = Counter(g for _, g, _, t in SAMPLE if g == t)
def pct(num, den):
    return f"{num / den:.0%}" if den else "n/a"


for cat in sorted(set(picked) | set(actual)):
    print(f"  {cat:<28} precision {pct(hits[cat], picked[cat]):>4}  "
          f"recall {pct(hits[cat], actual[cat]):>4}")

Output:

threshold  booked  precision  questions  booked but wrong
      0.0    100%        69%          0                5
      0.6     75%        83%          4                2
      0.8     69%        91%          5                1
      0.9     62%        90%          6                1
     0.95     50%       100%          8                0

per category, everything booked (threshold 0):
  Bank and payroll fees        precision 100%  recall  50%
  Contractors                  precision  50%  recall 100%
  Equipment expense            precision   0%  recall  n/a
  Fuel                         precision 100%  recall 100%
  Loan principal and interest  precision  n/a  recall   0%
  Materials                    precision  67%  recall 100%
  Owner contribution           precision  n/a  recall   0%
  Owner draw                   precision  n/a  recall   0%
  Payroll                      precision  67%  recall 100%
  Phone and internet           precision 100%  recall 100%
  Rent                         precision  n/a  recall   0%
  Sales                        precision  67%  recall 100%
  Utilities                    precision 100%  recall 100%

How do you read the threshold table?

Each line is a policy: "book anything at or above this confidence, ask about the rest."

• At 0.0 the AI books everything, and 5 of 16 are wrong. Those 5 sit in your books until someone notices.
• At 0.6 it books 75%, asks 4 questions, and 2 wrong rows still get booked.
• At 0.8 it books 69% and asks 5 questions. One wrong row slips through: ADP PAYROLL FEES booked as payroll at 0.91.
• At 0.95 it books half and asks 8 questions, and nothing booked is wrong.

Notice precision at 0.9 (90%) is lower than at 0.8 (91%). Raising the bar removed a correct row and kept the wrong one. Confidence isn't perfectly calibrated, which is exactly why you test on your own data instead of trusting the knob.

The column I'd watch is "booked but wrong". A question costs the owner a few seconds. A wrong booking costs whoever finds it later, usually at year end, usually billed by the hour.

What does the per-category table tell you?

Where the errors live. In this sample:

• The everyday categories are fine. Fuel, utilities, phone: 100% both ways.
• Payroll's precision is 67% because fees got lumped in with wages.
• Recall of 0% on loans, owner contributions and owner draws. The AI never used those categories. A loan payment booked as "Equipment expense" overstates expenses and leaves the loan balance wrong. A transfer from savings booked as "Sales" overstates revenue.
• Checks with no payee. One right, one wrong, both at low confidence. No model can know who check 3022 went to. Only the owner can.

That last group is the case for an AI that asks. The best behaviour on CHECK 3022 isn't a better guess. It's a question.

What should the AI do with what it isn't sure of?

The design that scores best on this test acts when it's sure, asks when it isn't, and logs everything so a wrong call can be undone. That's how ZeraBooks' Autopilot is built: it judges each item and says how sure it is, asks a one-tap question instead of guessing, and keeps every action in a log you can undo from. Categories come from each business's own reviewed history, so a correction to one Home Depot row can be applied to similar ones. And the ledger does every calculation; the model never does the arithmetic.

In a fictional run on one Bank of America statement for a landscaping company, ZeraBooks' Ask Zera imported 18 transactions, reconciled at a $0.00 difference, and left three items for the owner: a $5,000 transfer from savings and two checks with no payee. Those are the same kinds of rows that score 0% recall in the sample above. Leaving them as questions is the right answer.

ZeraBooks ranked seven AI bookkeepers on what each does alone, what it hands back to you, and whether you can undo it, in its list of the best AI bookkeeper tools. Booke AI, for example, says it leaves you exceptions to review; Puzzle says its AI drafts and you approve.

A test protocol you can run on any tool

1. Export two months of transactions with the AI's category and, if available, its confidence.
2. Have your accountant, or you on a careful day, label the correct category on every row.
3. Run the script. Look at "booked but wrong" at the tool's default setting.
4. Look at recall for transfers, loans, owner draws and contributions. Those are where real money gets misstated.
5. Check whether you can see and undo each AI action. If you can't, every wrong booking is a manual hunt.
6. Repeat after a month of corrections. A tool that learns from your history should do better on month three than month one.

If you're comparing whole AI accounting products rather than just their categorization, ZeraBooks' comparison of eight AI accounting tools lists each one's price and what its AI does, from each vendor's own pricing page.

Whatever you buy, keep the labelled sample. It's the only accuracy figure in this market that was measured on your books.