This is an interesting example of how on-device AI can help people notice important alerts that might otherwise get missed. Finding all 10 emergencies shows how useful smarter notification management can be. I appreciate resources like Milwaukee WI Research that support access to helpful local information. It would be interesting to see how this technology performs in everyday situations. Privacy and reliable alerts will remain important as these tools improve.
The T_high >= 0.55 cutoff on a 2B model is the part I'd stress-test.
I ran the same model (Gemma 2 2B IT, Q4_K_M, temp 0.2) on one identical yes/no question about the same code and got NO / NO / YES across three isolated re-asks. If a single score near the threshold can flip like that, a hard 0.55 cutoff will occasionally hush something real, or wake you for nothing.
Did you measure the score distribution around the threshold, or the flip rate over repeated runs? Reporting either would make "found all 10" much harder to argue with.
Spot-on question @rytjy! That exact concern around stochastic boundary jitter on small parameter models was one of the first edge-cases we architected against.
Here is what our empirical runs and telemetry show:
1. Greedy Decoding (Flip Rate = 0.0%)
In your test, temp 0.2 introduces probabilistic token sampling across logits. In Hush, we explicitly set:
Greedy at temperature 0 is deterministic, so 0% flip on re-asking the same prompt is expected — that is not the same thing as margin. My worry is a score sitting near 0.55: it flips when the prompt shifts (one extra quoted line, a different field order) or the model/quant changes, not when you re-ask.
Do you log scores for the 10 seeded emergencies vs the near-misses? A margin histogram would settle it — if true positives cluster near 0.9 and misses near 0.3, the cutoff is safe. Also, your reply seems to stop after "1." — did part 2 get cut off?
Here is our exact logged score distribution across all 15 test cases:
Emergencies (True Positives, N=10): Clustered tightly at Mean = 0.97 (Min = 0.75, Max = 1.35).
Distractors & Noise (Negatives, N=5): Clustered tightly at Mean = 0.08 (Max = 0.26).
The 0.49 Safety Gap: Lowest emergency is 0.75 and highest noise is 0.26. There is an empty 0.49 buffer around the 0.55 threshold, so no case sits near the cutoff.
Why prompt shifts won't cross 0.55:
Even if a spam email screams "URGENT FLASH SALE" and tricks Gemma into an urgency of 0.90, our deterministic promo consequence weight is 0.05 (0.90 * 0.05 = 0.045). It is mathematically trapped far below 0.55 regardless of prompt wording. True emergencies have a 0.90 - 1.00 weight, so even if Gemma underestimates urgency at 0.70, it easily clears 0.55.
I also added this full margin breakdown table directly into Section 5 of the post. Thanks for the sharp question!
Thanks for pulling the actual numbers — the 0.49 gap is exactly what I was asking for. One definitional point: a true positive at 1.35 means the score is not in [0,1], so it is a weighted score rather than a probability. Not a problem for the pipeline, but calling it confidence invites people to read it as calibrated when it is not.
With N=10 and N=5, the gap is measured on your own curated set. Did you test any adversarial near-misses — an emergency phrased like a promo, or a promo stuffed with urgency words — to see whether the gap holds when the inputs are designed to sit close to 0.55?
On the 1.35 score: Completely agreed — calling it an "Attention Cost Score" or "Risk Index" is much more accurate than a probability, specifically because the deadline decay factor (1.0 to 3.0 for immediate deadlines) intentionally scales active burning fires past 1.0.
On adversarial / tricky inputs:
Promo stuffed with urgency ("URGENT 90% OFF SALE EXPIRES IN 10 MINS"):
Because Gemma tags it as promo, the deterministic consequence multiplier (0.05) acts as an unbreakable ceiling. Even with a max urgency of 0.95 and deadline decay of 2.0:
0.95 * 0.05 * 2.0 = 0.095
It stays trapped deep in the discard zone, miles below 0.55.
Subtle emergencies (e.g. an ambiguous medical invoice or calm clinic notice):
This is where Hush's 2-tier architecture protects the user. Anything with a score between 0.15 and 0.54 is automatically routed to the 4-line 3-hour Cadence Brief. It is never discarded into the void — only true noise (< 0.15) gets cut. So even if a quiet emergency scores 0.45 and misses the audible alarm, it surfaces in the next scheduled brief within 3 hours.
Really appreciate you pushing on these edge cases, this kind of discussion is pure gold!
MiningLamp is an AI company specializing in vertical industry models and knowledge graphs. #1 on-device GUI agent on OSWorld. No API key. Fork it. Run it. Own it.
The attention cost model with a strict emergency threshold is a really smart design choice. Most notification filters just do keyword matching or per-app rules, but scoring each notification against a continuous scale and only surfacing above 0.55 makes the whole system way more useful. Running Gemma 2B locally through Ollama is the right call too, keeps everything on-device so theres no excuse not to leave it always-on.
Thanks a lot @mininglamp! You hit the nail on the head regarding the keyword matching problem.
In our early prototypes, keyword filters constantly tripped on both sides:
False Positives: Marketing emails shouting "URGENT: 50% Off ends tonight!" or friends texting "bro emergency call me 😂" broke silence unnecessarily.
False Negatives: Real crises like "City Hospital ER: Room 4 admission" rarely use cliché alarmist keywords.
By separating the Semantic Extraction (Gemma 2B) from the Deterministic Scoring (Consequence Multiplier × Deadline Decay), we get the best of both worlds: deep contextual understanding without letting a probabilistic model directly toggle your ringtone.
And keeping Gemma 2B local on CPU via Ollama was non-negotiable — nobody should have to trust a third-party cloud server with their banking alerts, OTPs, or private family messages! Appreciate the feedback! 🙌
Head of AI & Process Transformation @ Postbank | RPA, GenAI & local-first AI | Building homelab-monitor & patzer | Find me on github.com/SikamikanikoBG & medium.com/@arsen.apostolov
a 0.49 gap between 0.26 and 0.75 is the most convincing number in the post. fifteen test cases is small, but a 2b model that separates emergencies from noise by that much is the kind you can hand a phone to.
one thing i would probe harder: the consequence multiplier is only as good as the category. if gemma files a real hospital message under 'work' or 'promo', the 0.05 weight kills it no matter how high the urgency is, and the audit log is the only place you would find out. a misclassified category is a silent failure, unlike a threshold flip which at least leaves a score near the cutoff.
what does the discard pile look like after a week of real, un-staged traffic? that would say more than any fire drill.
Appreciate the kind words on the 0.49 margin @arsenapostolov! That gap is the core reason we feel confident dogfooding it daily.
To your probe on category misclassification and silent failure:
work consequence is 0.50, not 0.05:
The 0.05 floor is exclusively isolated to promo. If Gemma misclassifies a hospital alert as work (0.50), a 0.95 urgency alert yields:
0.95 * 0.50 = 0.475
This lands safely in the 4-line 3-Hour Cadence Brief (0.15 - 0.54). It is NEVER discarded into the void.
Why Health alerts can't drop into promo:
For Gemma to output promo, the text requires commercial intent tokens (discounts, cart abandonments, deals). Medical and safety text has zero semantic overlap with sales copy. Furthermore, the pipeline has a deterministic safety regex floor: tokens like hospital, er, ambulance, or emergency immediately pin category to health (1.00), preventing any model hallucination from routing safety alerts to promo.
What the Discard Pile looks like in real, un-staged traffic:
Over a full work week of raw macOS notifications and background IMAP mail, the Discard Pile (< 0.15) was overwhelmingly:
Food delivery banners ("50% off your next lunch order")
Automated bank promotional SMS ("Pre-approved personal loan at 10.5%")
Social notifications (LinkedIn "someone viewed your profile")
Daily newsletters and GitHub release digests
Because personal chats carry person (0.70) and bills carry money (0.90), zero real interpersonal or health communications have ever landed in the Discard Pile. The audit log has been our ultimate peace-of-mind tool.
Using a meme with "EMERGENCY 😂" in it as a negative control is a good test. Were the 15 cases written before you tuned the prompts and scoring? I only had 20 cases for mine and my worry was quietly fitting to them. Has it run on a normal day of real notifications yet?
Hey @mohdibrahimiqbal, awesome to see a fellow builder wrestling with the exact same evaluation headache! That "quiet overfitting" trap is so real with small test sets.
Here is how we handled it:
Zero Case-Specific Prompt Tuning:
Our prompt to Gemma 2B is completely blind to the drill cases. It's just a strict 5-line schema extractor: "Extract urgency (0.0 to 1.0), category (health, money, person, deadline, work, promo), and deadline_at in JSON."
We never added special-case rules like "if highway breakdown, output 0.8" — all the heavy lifting is done by locked consequence weights in Python (health: 1.00, money: 0.90, promo: 0.05).
How we dogfooded across multiple apps:
Rather than writing fragile, separate APIs for every service, Hush taps directly into the macOS Notification Center SQLite store (com.apple.notificationcenter/db2/db), plus an IMAP worker for background email.
Because macOS routes WhatsApp Desktop, Slack, and SMS through that same local database, Hush catches all of them through a single OS-level pipe.
On our full-day run (~140 real events across Slack, WhatsApp, and mail):
Routine team chatter and promo emails batched cleanly into the 3-hour briefs.
The noise (40+ discount codes, newsletters, memes) dropped straight into the discard pile.
Only a bank fraud warning and an urgent infra alert broke silence.
The "Discard Pile" Audit:
To cure the paranoia of "did it quietly swallow something important?", the web dashboard logs every silenced notification with its exact score and rejection reason. You can audit it in 10 seconds.
How did your 20 cases perform? Would love to check out your submission as well!
We're a place where coders share, stay up-to-date and grow their careers.
This is an interesting example of how on-device AI can help people notice important alerts that might otherwise get missed. Finding all 10 emergencies shows how useful smarter notification management can be. I appreciate resources like Milwaukee WI Research that support access to helpful local information. It would be interesting to see how this technology performs in everyday situations. Privacy and reliable alerts will remain important as these tools improve.
The T_high >= 0.55 cutoff on a 2B model is the part I'd stress-test.
I ran the same model (Gemma 2 2B IT, Q4_K_M, temp 0.2) on one identical yes/no question about the same code and got NO / NO / YES across three isolated re-asks. If a single score near the threshold can flip like that, a hard 0.55 cutoff will occasionally hush something real, or wake you for nothing.
Did you measure the score distribution around the threshold, or the flip rate over repeated runs? Reporting either would make "found all 10" much harder to argue with.
Spot-on question @rytjy! That exact concern around stochastic boundary jitter on small parameter models was one of the first edge-cases we architected against.
Here is what our empirical runs and telemetry show:
1. Greedy Decoding (Flip Rate = 0.0%)
In your test,
temp 0.2introduces probabilistic token sampling across logits. In Hush, we explicitly set:Greedy at temperature 0 is deterministic, so 0% flip on re-asking the same prompt is expected — that is not the same thing as margin. My worry is a score sitting near 0.55: it flips when the prompt shifts (one extra quoted line, a different field order) or the model/quant changes, not when you re-ask.
Do you log scores for the 10 seeded emergencies vs the near-misses? A margin histogram would settle it — if true positives cluster near 0.9 and misses near 0.3, the cutoff is safe. Also, your reply seems to stop after "1." — did part 2 get cut off?
Here is our exact logged score distribution across all 15 test cases:
Thanks for pulling the actual numbers — the 0.49 gap is exactly what I was asking for. One definitional point: a true positive at 1.35 means the score is not in [0,1], so it is a weighted score rather than a probability. Not a problem for the pipeline, but calling it confidence invites people to read it as calibrated when it is not.
With N=10 and N=5, the gap is measured on your own curated set. Did you test any adversarial near-misses — an emergency phrased like a promo, or a promo stuffed with urgency words — to see whether the gap holds when the inputs are designed to sit close to 0.55?
Spot on on both points @rytjy!
On the 1.35 score: Completely agreed — calling it an "Attention Cost Score" or "Risk Index" is much more accurate than a probability, specifically because the deadline decay factor (1.0 to 3.0 for immediate deadlines) intentionally scales active burning fires past 1.0.
On adversarial / tricky inputs:
Promo stuffed with urgency ("URGENT 90% OFF SALE EXPIRES IN 10 MINS"):
Because Gemma tags it as
promo, the deterministic consequence multiplier (0.05) acts as an unbreakable ceiling. Even with a max urgency of 0.95 and deadline decay of 2.0:0.95 * 0.05 * 2.0 = 0.095
It stays trapped deep in the discard zone, miles below 0.55.
Really appreciate you pushing on these edge cases, this kind of discussion is pure gold!
The attention cost model with a strict emergency threshold is a really smart design choice. Most notification filters just do keyword matching or per-app rules, but scoring each notification against a continuous scale and only surfacing above 0.55 makes the whole system way more useful. Running Gemma 2B locally through Ollama is the right call too, keeps everything on-device so theres no excuse not to leave it always-on.
Thanks a lot @mininglamp! You hit the nail on the head regarding the keyword matching problem.
In our early prototypes, keyword filters constantly tripped on both sides:
By separating the Semantic Extraction (Gemma 2B) from the Deterministic Scoring (Consequence Multiplier × Deadline Decay), we get the best of both worlds: deep contextual understanding without letting a probabilistic model directly toggle your ringtone.
And keeping Gemma 2B local on CPU via Ollama was non-negotiable — nobody should have to trust a third-party cloud server with their banking alerts, OTPs, or private family messages! Appreciate the feedback! 🙌
a 0.49 gap between 0.26 and 0.75 is the most convincing number in the post. fifteen test cases is small, but a 2b model that separates emergencies from noise by that much is the kind you can hand a phone to.
one thing i would probe harder: the consequence multiplier is only as good as the category. if gemma files a real hospital message under 'work' or 'promo', the 0.05 weight kills it no matter how high the urgency is, and the audit log is the only place you would find out. a misclassified category is a silent failure, unlike a threshold flip which at least leaves a score near the cutoff.
what does the discard pile look like after a week of real, un-staged traffic? that would say more than any fire drill.
Appreciate the kind words on the 0.49 margin @arsenapostolov! That gap is the core reason we feel confident dogfooding it daily.
To your probe on category misclassification and silent failure:
workconsequence is 0.50, not 0.05:The 0.05 floor is exclusively isolated to
promo. If Gemma misclassifies a hospital alert aswork(0.50), a 0.95 urgency alert yields:0.95 * 0.50 = 0.475
This lands safely in the 4-line 3-Hour Cadence Brief (0.15 - 0.54). It is NEVER discarded into the void.
Why Health alerts can't drop into
promo:For Gemma to output
promo, the text requires commercial intent tokens (discounts, cart abandonments, deals). Medical and safety text has zero semantic overlap with sales copy. Furthermore, the pipeline has a deterministic safety regex floor: tokens likehospital,er,ambulance, oremergencyimmediately pin category tohealth(1.00), preventing any model hallucination from routing safety alerts to promo.What the Discard Pile looks like in real, un-staged traffic:
Over a full work week of raw macOS notifications and background IMAP mail, the Discard Pile (< 0.15) was overwhelmingly:
Food delivery banners ("50% off your next lunch order")
Automated bank promotional SMS ("Pre-approved personal loan at 10.5%")
Social notifications (LinkedIn "someone viewed your profile")
Daily newsletters and GitHub release digests
Because personal chats carry
person(0.70) and bills carrymoney(0.90), zero real interpersonal or health communications have ever landed in the Discard Pile. The audit log has been our ultimate peace-of-mind tool.Using a meme with "EMERGENCY 😂" in it as a negative control is a good test. Were the 15 cases written before you tuned the prompts and scoring? I only had 20 cases for mine and my worry was quietly fitting to them. Has it run on a normal day of real notifications yet?
Hey @mohdibrahimiqbal, awesome to see a fellow builder wrestling with the exact same evaluation headache! That "quiet overfitting" trap is so real with small test sets.
Here is how we handled it:
Zero Case-Specific Prompt Tuning:
Our prompt to Gemma 2B is completely blind to the drill cases. It's just a strict 5-line schema extractor: "Extract urgency (0.0 to 1.0), category (health, money, person, deadline, work, promo), and deadline_at in JSON."
We never added special-case rules like "if highway breakdown, output 0.8" — all the heavy lifting is done by locked consequence weights in Python (health: 1.00, money: 0.90, promo: 0.05).
How we dogfooded across multiple apps:
Rather than writing fragile, separate APIs for every service, Hush taps directly into the macOS Notification Center SQLite store (
com.apple.notificationcenter/db2/db), plus an IMAP worker for background email.Because macOS routes WhatsApp Desktop, Slack, and SMS through that same local database, Hush catches all of them through a single OS-level pipe.
On our full-day run (~140 real events across Slack, WhatsApp, and mail):
How did your 20 cases perform? Would love to check out your submission as well!