I write $0-stack engineering guides: running local LLMs (DeepSeek, Qwen, Llama) on consumer laptops, and shipping full-stack apps on free tiers. Everything benchmarked on real hardware.
Fascinating deep-dive, @xbill! The zero-card, points-budget model of AMD Token Lab is a massive unlock for developers prototyping ADK agents without cloud burn.
Two architectural questions/notes from running similar agent loops on open-weights:
ADK Agent Turn Latency & Prefix Caching: When running an agent loop with tool declarations, re-sending the system prompt and tool definitions on every turn eats through points budgets fast. Does Token Lab's SGLang backend preserve KV prefix caching across multi-turn agent calls? In our testing, SGLang radix attention/prefix caching reduces time-to-first-token (TTFT) by ~8x on 31B parameter models.
Context Compaction for Agent Tool Catalogs: For local agent rigs, lazy tool schemas (or Code Mode runtimes) drastically reduce input tokens compared to dumping full JSON schemas into Gemma's context window.
We recently published a companion hardware analysis on consumer GPU allocation and avoiding silent CPU fallback: dev.to/thezerodollardev/your-8gb-g...
Great to see AMD and GDE highlighting zero-dollar developer tiers that are genuinely production-capable!
I write $0-stack engineering guides: running local LLMs (DeepSeek, Qwen, Llama) on consumer laptops, and shipping full-stack apps on free tiers. Everything benchmarked on real hardware.
100% on Gemma—its architectural balance between instruction accuracy and lightweight token footprint makes it really snappy for ADK state loops.
Have you tried running Qwen 2.5 Coder or DeepSeek on AMD Token Lab yet? In our testing across budget developer environments, Qwen 2.5 (especially 7B and 14B) shows remarkably low hallucination rates on rigid JSON schemas compared to standard instruction models.
Also, AMD Token Lab not forcing a credit card upfront is huge. That friction-free setup lets solo developers test multi-turn agent prototypes without unexpected billing surprises.
I usually stick to gemma models for consistency - but the approach should work with any model they offer. putting it on my todo list to sample a couple of models
I write $0-stack engineering guides: running local LLMs (DeepSeek, Qwen, Llama) on consumer laptops, and shipping full-stack apps on free tiers. Everything benchmarked on real hardware.
That makes total sense—Gemma's prompt format and instruction-following are super reliable across tasks. If you do test others on Token Lab, Qwen 2.5 Coder (14B/32B) is particularly interesting for code/schema generation, and DeepSeek-R1 Distill for multi-step reasoning steps. Looking forward to reading your follow-up benchmarks!
I write $0-stack engineering guides: running local LLMs (DeepSeek, Qwen, Llama) on consumer laptops, and shipping full-stack apps on free tiers. Everything benchmarked on real hardware.
Totally agree on Gemma—its architectural balance between instruction-following and token economy makes it exceptionally responsive, especially for ADK agent state management.
Curious if you've experimented with Qwen 2.5 Coder or DeepSeek on the Token Lab infrastructure yet? In our benchmarks on resource-constrained setups, Qwen 2.5 7B/14B exhibits remarkably low hallucination rates on structured JSON tool calling compared to traditional instruction models.
Also love that AMD Token Lab doesn't mandate credit card pre-auth—that zero-barrier tier is exactly what independent developers need to validate multi-turn agent loops before committing to paid API tiers.
A technologist working in automation. Here I write up what I actually learn building tools, engineering AI agents, and keeping systems running, along with the workflows and habits that stick.
Checking for data: [DONE] in chat_ui.py catches a gap that num_retries=30 misses. LiteLLM retries HTTP status codes like 429 or 503 before the response headers arrive. If a backend restart drops an open SSE stream mid-generation, the agent still gets a truncated reply unless the client detects the missing terminal marker and restarts the turn.
this one is a bit of a black box. my probes say its a sglang router- so maybe the external endpoint is some kind of gateway. which makes sense as you would need routing plus DOS protection
I research AI tools, APIs & agents — comparing specs, pricing and real user reports — then publish honest rankings so you don't waste money. Research-first, never sponsored fluff.
The timing table is the bit that stuck with me — 7.55 tok/s on one run, 3.41 on another, same model, same laptop. That's the real price of 'free' here, isn't it? Not points, you barely touched 0.000335 of your 10, but not knowing whether your next request lands in a quiet moment or behind 127 other people. I liked the [DONE] check too, that's the sort of thing you only add after a stream dies halfway through something you actually needed.
Thanks for publishing actual numbers instead of “it works great.” One thing they show clearly is that the 10-point budget is almost irrelevant. 23 requests used 0.000335 points, and by my rough math, a single stream decoding nonstop at your measured 7.5 tok/s for all 30 days would produce about 19.6M tokens, or roughly 1.4 points at the output rate. You’d need around seven streams running around the clock to exhaust the allowance. So the real limits are the 20 requests/minute cap, the shared 128-request ceiling, and decode speed. That changes how I’d design on top of it.
Think about retries against the rate limit. num_retries=30 on the ADK agent is generous, but retries presumably count toward your 20 RPM. An agent loop makes one request per tool round trip, so a multi-step task plus retries can burn through the budget quickly and then retry into the 429s it created. A small client-side token bucket (e.g. 15 requests/minute), exponential backoff with jitter on top of Retry-After, and a cap on total retries per turn would behave better than a flat 30.
Consider stream=False for the agent. Your tip that a stream broken mid-reply can’t be retried automatically is the sharpest edge here. A chat page needs streaming, but an agent usually doesn’t. A non-streamed request either returns a complete message or fails cleanly, so it can be retried as a unit. That also removes the : ping, id: and event: done parsing problems. For tool-calling turns this is especially convenient, since tools only execute after the full response arrives.
Thinking is a poor trade on this endpoint. In your table, 291 of 390 tokens were reasoning, so about three quarters of the output was spent before the answer, at 1.98 tok/s, for a 200 s total. Disabling thinking in the agent via enable_thinking: False looks like the right default. Thinking could be an opt-in for specific hard questions, perhaps through a separate agent or model object.
The timing table hints at queueing rather than compute. The 256-token request got its first token in 1.22 s, while the 97-token request took 5.04 s. First-token latency didn’t track prompt or output size, which fits your point about shared load. It also shows in the agent trace: list_tokenlab_models took roughly 13 s between call and result, even though that’s just a catalog fetch. If you rerun this, I’d log p50/p95 over a few dozen requests at different hours. Four data points tell the story qualitatively but can’t support planning.
A localhost proxy with the key attached deserves a guard. Keeping the key out of the browser is the right call. But a server on 127.0.0.1 that forwards /v1/* with your credentials will serve any request that reaches it, including ones triggered by a web page you happen to have open (a plain cross-site POST, or DNS rebinding). Cheap mitigations: reject requests whose Host isn’t localhost:8766, check Origin on POSTs, and allowlist only /v1/chat/completions and /v1/models. That keeps the “free” key from being spent by someone else’s page.
A nice pattern in the agent. count_and_range returning the filter string alongside the count is a small design choice that matters: the model’s interpretation of “10 or more” becomes visible and auditable (10 <= x), so a wrong parameter mapping shows up in the answer instead of hiding in a plausible number. I’d apply the same habit to any tool where the model chooses boundaries or units.
Finally, since this is a shared free service, check the data-handling terms before sending anything proprietary through it. For prototyping on public or synthetic data it looks like a great fit, which matches your conclusion.
the count_and_range split is a useful boundary: code does the exact filtering, then the model has to expose the filter it applied. did you try ambiguous wording where that interpretation could quietly shift?
Your timing table is the part I keep coming back to. The same model at 7.55 tokens per second on one run and 3.41 on another makes the shared endpoint point in a way no marketing line could. I also liked the practical detail about keeping the key on the local machine and forwarding through the chat page, so the browser never sees it. That is the kind of setup note I wish more free tier guides included.
We're a place where coders share, stay up-to-date and grow their careers.
Fascinating deep-dive, @xbill! The zero-card, points-budget model of AMD Token Lab is a massive unlock for developers prototyping ADK agents without cloud burn.
Two architectural questions/notes from running similar agent loops on open-weights:
We recently published a companion hardware analysis on consumer GPU allocation and avoiding silent CPU fallback: dev.to/thezerodollardev/your-8gb-g...
Great to see AMD and GDE highlighting zero-dollar developer tiers that are genuinely production-capable!
I like the low risk/no cost approach. I went with Gemma because I like working with that model- but there are tons of options you can test drive.
100% on Gemma—its architectural balance between instruction accuracy and lightweight token footprint makes it really snappy for ADK state loops.
Have you tried running Qwen 2.5 Coder or DeepSeek on AMD Token Lab yet? In our testing across budget developer environments, Qwen 2.5 (especially 7B and 14B) shows remarkably low hallucination rates on rigid JSON schemas compared to standard instruction models.
Also, AMD Token Lab not forcing a credit card upfront is huge. That friction-free setup lets solo developers test multi-turn agent prototypes without unexpected billing surprises.
I usually stick to gemma models for consistency - but the approach should work with any model they offer. putting it on my todo list to sample a couple of models
That makes total sense—Gemma's prompt format and instruction-following are super reliable across tasks. If you do test others on Token Lab, Qwen 2.5 Coder (14B/32B) is particularly interesting for code/schema generation, and DeepSeek-R1 Distill for multi-step reasoning steps. Looking forward to reading your follow-up benchmarks!
Totally agree on Gemma—its architectural balance between instruction-following and token economy makes it exceptionally responsive, especially for ADK agent state management.
Curious if you've experimented with Qwen 2.5 Coder or DeepSeek on the Token Lab infrastructure yet? In our benchmarks on resource-constrained setups, Qwen 2.5 7B/14B exhibits remarkably low hallucination rates on structured JSON tool calling compared to traditional instruction models.
Also love that AMD Token Lab doesn't mandate credit card pre-auth—that zero-barrier tier is exactly what independent developers need to validate multi-turn agent loops before committing to paid API tiers.
Checking for
data: [DONE]inchat_ui.pycatches a gap thatnum_retries=30misses. LiteLLM retries HTTP status codes like429or503before the response headers arrive. If a backend restart drops an open SSE stream mid-generation, the agent still gets a truncated reply unless the client detects the missing terminal marker and restarts the turn.this one is a bit of a black box. my probes say its a sglang router- so maybe the external endpoint is some kind of gateway. which makes sense as you would need routing plus DOS protection
The timing table is the bit that stuck with me — 7.55 tok/s on one run, 3.41 on another, same model, same laptop. That's the real price of 'free' here, isn't it? Not points, you barely touched 0.000335 of your 10, but not knowing whether your next request lands in a quiet moment or behind 127 other people. I liked the [DONE] check too, that's the sort of thing you only add after a stream dies halfway through something you actually needed.
I think its mostly about expectations- for a no cost/low risk basic validation - 7.55 tok/s is fine. For anything else- painfully slow.
Thanks for publishing actual numbers instead of “it works great.” One thing they show clearly is that the 10-point budget is almost irrelevant. 23 requests used 0.000335 points, and by my rough math, a single stream decoding nonstop at your measured 7.5 tok/s for all 30 days would produce about 19.6M tokens, or roughly 1.4 points at the output rate. You’d need around seven streams running around the clock to exhaust the allowance. So the real limits are the 20 requests/minute cap, the shared 128-request ceiling, and decode speed. That changes how I’d design on top of it.
Think about retries against the rate limit. num_retries=30 on the ADK agent is generous, but retries presumably count toward your 20 RPM. An agent loop makes one request per tool round trip, so a multi-step task plus retries can burn through the budget quickly and then retry into the 429s it created. A small client-side token bucket (e.g. 15 requests/minute), exponential backoff with jitter on top of Retry-After, and a cap on total retries per turn would behave better than a flat 30.
Consider stream=False for the agent. Your tip that a stream broken mid-reply can’t be retried automatically is the sharpest edge here. A chat page needs streaming, but an agent usually doesn’t. A non-streamed request either returns a complete message or fails cleanly, so it can be retried as a unit. That also removes the : ping, id: and event: done parsing problems. For tool-calling turns this is especially convenient, since tools only execute after the full response arrives.
Thinking is a poor trade on this endpoint. In your table, 291 of 390 tokens were reasoning, so about three quarters of the output was spent before the answer, at 1.98 tok/s, for a 200 s total. Disabling thinking in the agent via enable_thinking: False looks like the right default. Thinking could be an opt-in for specific hard questions, perhaps through a separate agent or model object.
The timing table hints at queueing rather than compute. The 256-token request got its first token in 1.22 s, while the 97-token request took 5.04 s. First-token latency didn’t track prompt or output size, which fits your point about shared load. It also shows in the agent trace: list_tokenlab_models took roughly 13 s between call and result, even though that’s just a catalog fetch. If you rerun this, I’d log p50/p95 over a few dozen requests at different hours. Four data points tell the story qualitatively but can’t support planning.
A localhost proxy with the key attached deserves a guard. Keeping the key out of the browser is the right call. But a server on 127.0.0.1 that forwards /v1/* with your credentials will serve any request that reaches it, including ones triggered by a web page you happen to have open (a plain cross-site POST, or DNS rebinding). Cheap mitigations: reject requests whose Host isn’t localhost:8766, check Origin on POSTs, and allowlist only /v1/chat/completions and /v1/models. That keeps the “free” key from being spent by someone else’s page.
A nice pattern in the agent. count_and_range returning the filter string alongside the count is a small design choice that matters: the model’s interpretation of “10 or more” becomes visible and auditable (10 <= x), so a wrong parameter mapping shows up in the answer instead of hiding in a plausible number. I’d apply the same habit to any tool where the model chooses boundaries or units.
Finally, since this is a shared free service, check the data-handling terms before sending anything proprietary through it. For prototyping on public or synthetic data it looks like a great fit, which matches your conclusion.
the count_and_range split is a useful boundary: code does the exact filtering, then the model has to expose the filter it applied. did you try ambiguous wording where that interpretation could quietly shift?
Your timing table is the part I keep coming back to. The same model at 7.55 tokens per second on one run and 3.41 on another makes the shared endpoint point in a way no marketing line could. I also liked the practical detail about keeping the key on the local machine and forwarding through the chat page, so the browser never sees it. That is the kind of setup note I wish more free tier guides included.