On a machine without a GPU, Ollama 0.40.14 loaded qwen3-embedding:0.6b with a 4,096-token context, and /api/embed cut my 7,510-token document down to 4,095 tokens with no error, no warning and no flag in the response. Send "truncate": false so an overlong input fails with HTTP 400, and split documents into chunks of about 2,000 characters before you embed them.
This story was written with the assistance of an AI writing program. Every number below comes from one run on an 8-core Linux box with no GPU: Ollama 0.40.14, qwen3-embedding:0.6b (Q8_0, 1,024 dimensions) and Python 3.14.4 calling the HTTP API with urllib. The test document is the GPL-3 license text (35,149 characters).
The Model Says 32,768, the Server Uses 4,096
/api/show reports qwen3.context_length = 32768 for this model. That is what the model was trained for, not what the server used. After the first request, /api/ps listed the loaded model with context_length=4096, and the server log said why: msg="vram-based default context" total_vram="0 B" default_num_ctx=4096.
With no VRAM to measure, the server picked 4,096 tokens. Embedding requests run in that same context, so one embedding sees at most 4,095 tokens of input.
A Sentence at the End Changed Nothing
I wrote one fact that appears nowhere in the license: "The staging cluster's nightly backup runs at 02:30 UTC and keeps 14 snapshots." Then I embedded three documents: the license alone, the license with the fact appended at the end, and the license with the fact at the start. The query was "When does the staging cluster's nightly backup run?"
D_none (GPL only): HTTP 200 44.6s chars=35149 prompt_eval_count=4095 keys=['embeddings', 'load_duration', 'model', 'prompt_eval_count', 'total_duration']
D_end (needle last): HTTP 200 50.6s chars=35230 prompt_eval_count=4095 keys=['embeddings', 'load_duration', 'model', 'prompt_eval_count', 'total_duration']
e_end == e_none (exact list equality): True max|diff| = 0.0
cos(q, D_none) = 0.2773 cos(q, D_end) = 0.2773 cos(q, D_start) = 0.7830
The vector for "license plus fact at the end" is identical to the vector for the license alone: all 1,024 floats equal. Both score 0.2773 against the query. Put the same sentence first and the score jumps to 0.7830. In a RAG index this is the quiet kind of failure: the chunk is stored, the vector looks normal, and nothing in the second half of the text can ever be retrieved.
Nothing in the Response Says It Was Cut
The response has five keys, and none of them is a truncation flag. The only hint is prompt_eval_count = 4095, the context minus one, and you only notice it if you compare it with what you sent. The runner's log line for the same request reads n_tokens = 4095, truncated = 0, so grepping the server log for truncation finds nothing either.
The behaviour is documented. The API reference says truncate "truncates the end of each input to fit within context length. Returns error if false and context length is exceeded. Defaults to true". The default is the trap. At this text's 4.7 characters per token, everything after roughly the first 19,000 characters was dropped.
truncate=false Turns It Into a 400
This is the guard I now put in front of every embedding call:
import json, urllib.request
def embed(texts, url="http://127.0.0.1:11434"):
body = {"model": "qwen3-embedding:0.6b", "input": texts, "truncate": False}
req = urllib.request.Request(url + "/api/embed", data=json.dumps(body).encode(),
headers={"Content-Type": "application/json"})
with urllib.request.urlopen(req) as r: # raises HTTPError 400 instead of truncating
return json.loads(r.read())["embeddings"]
A short input still returns a 1,024-dimension vector. The full license raised HTTPError 400 after 0.03 seconds with {"error":"the input length exceeds the context length"}. That is the failure you want: fast, loud, and before a useless vector lands in the index. The silent version took 44.6 seconds to produce that useless vector.
Chunks Found the Fact; the Long Context Barely Did
The fix that worked was splitting the text on paragraph boundaries into chunks of at most 2,000 characters. That gave 20 chunks, the longest 1,992 characters, sent as one batch with truncate: false. The batch returned HTTP 200 in 54.1 seconds for 7,529 tokens. The chunk holding the fact ranked first of 20 with a cosine of 0.8372; the next best scored 0.3272.
The other obvious fix is a bigger context. I started a second server with OLLAMA_CONTEXT_LENGTH=16384 and embedded the whole document in one call. All 7,510 tokens were read this time, and the fact did move the vector: the query scored 0.6103 against the license with the fact and 0.3650 without it. That is retrievable, but far weaker than 0.8372, because one vector now has to summarise 7,500 tokens of license text plus one sentence about backups.
What a Bigger Context Costs on CPU
The single 7,510-token embedding took 124.0 seconds. The 20 chunks covering the same text took 54.1 seconds, 2.3 times faster. The loaded model grew too: /api/ps reported 1.24 GB at a 4,096-token context, 1.72 GB at 8,192 and 2.70 GB at 16,384. A per-request "options": {"num_ctx": 8192} also raised the context, but the server reloaded the model to apply it, so mixing context sizes across requests means reloads.
I did not test a GPU machine; its default context may be larger, but the trap is the same whenever a document is longer than the loaded context.
Takeaways
- Do not trust the model card's context length. Check
/api/psor the server log for the context that was actually loaded. - Set
"truncate": falseon every/api/embedcall. A 400 in hundredths of a second beats a vector that ignores half the document. - Log
prompt_eval_countnext to each input's length. A value equal to the context minus one means the input was cut. - Chunk before you embed. Here, 2,000-character chunks found the fact at 0.84 cosine and ran 2.3 times faster than one embedding of the whole text at a 16,384-token context.
- If you do raise the context, budget for it: 124 seconds for one document, and the model grew from 1.24 GB to 2.70 GB.
Top comments (0)