DEV Community

Cover image for The AI wrote a page from my notes, and I can't tell which note a sentence came from

The AI wrote a page from my notes, and I can't tell which note a sentence came from

Masaya Kumagai on November 13, 2026

Say you have an AI gather your notes and the things you read into pages, one per subject. Pointing an AI at a folder of Markdown notes and letting ...
Collapse
 
izgorodin profile image
Edward Izgorodin •

Attaching the source at the moment of writing is what makes the link a record, and the revision after it is where that record gets tested. Say a later trial contradicts the 65% sentence. The AI can keep the old sentence with its old link, rewrite it and cite both notes, or drop it, so that only the page history remembers what the earlier note said. Each of those tells a reader something different about what is still true. Which one does Graphium do today, and when a sentence is dropped, does its note leave the References list as well?

Collapse
 
kumagallium profile image
Masaya Kumagai •

Good question, and checking it showed me a gap. When a later source contradicts the 65% sentence, Graphium keeps both: the old sentence with its old citation, and the new figure as a separate sentence with its own. It lists the disagreement under “Disagreements & open questions” and doesn’t decide which one is true. A sentence is rewritten or dropped only when the same note is re-read and no longer says it.

The gap was that “keep both” lived only in the instructions to the model. Nothing checked it. Now Graphium compares the citations before and after each revision. If every sentence citing a note it didn’t re-read has disappeared, it asks the model once more to keep them.

On References: until now a note stayed in the list even when no sentence cited it any more, so the list looked like it backed more than the text did. The note still stays, because it was read and the provenance should say so. But it now moves to the end, marked “read, but not cited in the current text”. The page history keeps what the earlier version said. In the next release, each sentence will also carry the passage of the note it was written from, so you can see exactly what the earlier note said.

Collapse
 
izgorodin profile image
Edward Izgorodin •

Moving an uncited note to the end with that label is a good call: the list stops overstating the text, and nothing that was read disappears. One thing the Disagreements section could carry cheaply is the date of each note next to its figure. Without it, a reader sees 65% and the newer figure side by side and cannot tell which came later, and order in time is often the first question a reader asks, even though it settles nothing about which figure is right. Will the per-sentence passage in the next release stay as it was when the note itself is edited later?

Thread Thread
 
kumagallium profile image
Masaya Kumagai •

Both are in now. Under “Disagreements & open questions”, each figure’s citation is followed by its source’s date: the note’s creation date, or the paper’s year when Graphium knows it. As you say, it doesn’t settle which figure is right, but it answers the first question a reader asks.

On edits: the passage stays as it was until the note is read again. When it is, every passage is checked against the note’s current full text. If the note no longer says it, the passage isn’t removed. It stays with a mark, and the sentence shows “the passage the AI copied isn’t in the source”, so the page records what it was written from and that the note has changed since. The note’s own history keeps the earlier wording. If a later version says it again, the mark goes away.

Thread Thread
 
izgorodin profile image
Edward Izgorodin •

Keeping the mark instead of removing the passage is the right default, and the note history makes it recoverable. One gap I would check is the time between an edit and the next read. If a note can change without being read again, for example when it is edited in another app, the page keeps showing its sentences as supported until someone triggers a read, and for a note nobody reopens that can be weeks. Storing a hash of the note text with each passage would let the page compare on open and show that the source changed since it was written, before the full passage check runs.

Collapse
 
kolmopdf profile image
KolmoPDF •

Attaching the source link during synthesis instead of running a post-hoc search fixes the nastiest failure mode in LLM wikis. Once a topic page goes through three or four revisions, a backward attribution pass almost always matches a plausible neighboring note rather than the file the model actually read.

One boundary worth watching if you ever extend Graphium beyond plain Markdown notes into PDFs or research papers is chunk-level provenance before synthesis. When a note comes from a multi-page PDF, page-level links alone get painful to audit if you have to hunt through a 30-page paper to check one number. Carrying section headings and table captions down into the extracted Markdown blocks lets the sentence link point to a specific heading or page anchor, and keeps your separate source-check feature from flagging numbers that simply lost their table headers during extraction.

The explicit merge prompt for split topics is a smart guardrail too. Silent auto-merging between near-duplicate subjects like spelling variants tends to corrupt long-running knowledge bases faster than leaving two separate pages around for a human to reconcile.

Collapse
 
kumagallium profile image
Masaya Kumagai •

Thank you, and yes, that matches what I saw: after a few revisions, the “nearest” note is usually the one a backward search finds.

Graphium already takes PDFs, Word files and web pages as sources, not just notes. Your point about where inside the source holds, though. When you run the source check on a page, it tries to show where each quote sits in a PDF, such as “Page 12”. It only does this when the model’s quote appears word for word in the extracted text, and it leaves the line out when the same wording appears on more than one page. I haven’t yet measured how often it lands on the right page with real papers, where two-column layouts and headers make extraction messy. The sentence link in a topic, on the other hand, still points to the whole document. Carrying the page or heading down to the sentence itself would make checking much easier, and I’ll keep your note about table captions in mind for that.

Collapse
 
paul_irolla_6b5ae261d0224 profile image
Paul Irolla •

A string match between each sentence and the note it cites catches most invented links before a second model has to judge them. It's cheap, and it's deterministic, so a failure can be reproduced.

Collapse
 
kumagallium profile image
Masaya Kumagai •

Thanks, this pushed me to build it. One snag in Graphium: the sentences in a topic are paraphrased, so matching a sentence directly against its note rarely hits. So I moved the string match one step earlier. When the model writes a sentence, it now also copies, inside the citation, the passage of the source it is restating, word for word. Before the page is saved, Graphium searches the source text for that passage. No second model, and the same result every time. If the passage isn’t there, it is dropped, and that sentence goes to the source check like any other. In a first measurement with an open model, 99% of the passages were found in the source. The one that wasn’t had a word “corrected” by the model. A side effect I like: the citation can now open a PDF at that exact passage instead of at page 1. It ships in the next release.

Collapse
 
paul_irolla_6b5ae261d0224 profile image
Paul Irolla •

@kumagallium thanks, glad it pushed you to build it. Checking the passage at write time is the right place for it: a verbatim substring test gives the same answer on every run, so a failure can be reproduced. I would keep the 1% miss visible rather than fix it quietly. A word the model corrected is exactly what a reader should see, so that sentence should fall to the source check.

Collapse
 
kumagallium profile image
Masaya Kumagai •

Agreed, and it’s in now. A passage that isn’t found in the source is no longer removed. It stays on the page with a mark, and the sentence shows “the passage the AI copied isn’t in the source” right after its citation. The source check details show the passage the model actually wrote, next to the verdict, so you can see which word it changed. Passages are also checked against the whole source now, not just the window the model read. That catches a second case quietly fixing would have hidden: a note edited after the page was written.