DEV Community

Andrey Altrouter
Andrey Altrouter

Posted on

A third of the tokens in the code you send an LLM are comments. Indentation is 2.5%.

Sooner or later every coding-assistant project hits the same idea: the files we paste into the prompt are big, let's minify them. Collapse the indentation, delete blank lines, squeeze the whitespace like a JavaScript bundler does. It feels like the obvious win, because code looks like it's half spaces.

I measured it. The whitespace is almost free. The expensive part of a source file is the English inside it.

A few words first

A token is a chunk of text the model reads as one piece; APIs bill per million of them. Input tokens are what you send, output tokens are what the model writes back, and each has its own price. A tokenizer is the program that cuts text into tokens. I used tiktoken, OpenAI's open-source one, with o200k_base, the encoding behind current GPT models. A docstring is the string literal at the top of a Python module, class or function that documents it; a comment is anything after #.

What I counted

The corpus is every top-level .py module of the CPython 3.13 standard library: 131 files, 107,041 lines, 893,573 tokens. I skipped test files and private underscore modules. That's real, maintained, widely read code, and the kind of thing a model sees a lot of.

Then I stripped one layer at a time, with Python's own tokenize and ast modules, so the result still parses, and counted what each layer cost:

Part of the file Tokens Share
Docstrings 188,914 21.1%
# comments 122,164 13.7%
Indentation (4 spaces → 1) and blank lines 14,661 1.6%
Code that's left 567,834 63.5%

Comments and docstrings together are 34.8% of the tokens. Re-indenting the untouched originals from four spaces to one saves 2.5%; deleting every blank line saves 0.1%. In the middle file, prose is 34.1% of the tokens. One file in ten is under 15.4%, and one in ten is over 51.4%.

Why the spaces are cheap

The tokenizer packs a run of spaces into a single token. A line indented by eight spaces comes out as ' ', ' return', ' x', '\n': one token of indentation, however deep the block is. Shrinking it to one space just moves that space into the ' return' token, so you save one token per indented line, not one per space. Blank lines cost even less: deleting all of them saved 558 tokens across the whole corpus. Meanwhile a docstring is ordinary English, roughly one token per word, and the stdlib writes a lot of it.

You can check any file of your own in a dozen lines:

import ast, inspect, textwrap
import tiktoken

enc = tiktoken.get_encoding("o200k_base")
src = inspect.getsource(textwrap)          # or open("your_file.py").read()

tree = ast.parse(src)
for node in ast.walk(tree):
    body = getattr(node, "body", None)
    if (isinstance(body, list) and body and isinstance(body[0], ast.Expr)
            and isinstance(body[0].value, ast.Constant)
            and isinstance(body[0].value.value, str)):
        node.body = body[1:] or [ast.Pass()]   # drop the docstring
lean = ast.unparse(tree)                       # unparse also drops every # comment

print(len(enc.encode(src)), "->", len(enc.encode(lean)), "tokens")
Enter fullscreen mode Exit fullscreen mode

On textwrap.py it prints 4491 -> 1893 tokens, 58% less. ast.unparse rebuilds the file from the syntax tree, and the tree has no place to keep # comments, so they disappear on their own.

What it's worth

Say an assistant sends three files of context with every request, each the size of the median stdlib module (4,250 tokens), 1,000 requests a day. At the official list prices from the catalog of the LLM gateway I work on (snapshot of 14 November 2026), GPT-5.6 and Claude Sonnet 5.5 both charge $2 per million input tokens. That's $765 a month just for the code, and at the stdlib's 34.8% the comments and docstrings are $266 of it. On Claude Haiku 5.5 at $0.10 per million it's $38 a month, $13 of it prose. Collapsing the indentation instead would save about $19 on the bigger models.

The percentages don't depend on the vendor. Price changes the dollar amount, and the share of prose in your files sets the percentage.

What this doesn't solve

Comments are there for a reason, and the model reads them too. A docstring that says "returns None if the user is not found" is exactly what the model needs to use that function correctly. Strip it and you may save a third of the tokens and lose the right answer. I measured tokens, not answer quality, so I can't tell you how often that happens.

The safe split is by role. The file the model is asked to change should stay intact: if you strip it, the model's edits refer to lines that don't exist in your real file, and its output may quietly drop the comments you wanted to keep. The files that are only there as background are the ones to slim down.

Your code isn't the standard library either. The stdlib documents itself heavily; plenty of application code has almost no docstrings, and then there's little to strip. Run the snippet on your own repo before you build anything. The counts come from OpenAI's tokenizer, and Anthropic and Google count a few percent differently. If you resend the same files on every call, prompt caching can make them cheaper still, and it may beat stripping. Nothing in the API layer can do this for you, including the gateway I work on: it sees tokens, not which comments matter.

The takeaway

Before you write a minifier for your prompts, count where the tokens are. In Python they're in the docstrings and comments, about a third of the file, and the whitespace you were going to collapse is 2.5%.

Top comments (0)