'Don't reveal it' is an instruction, and an instruction is a request, not a control. I run agents all day and this is the same lesson in a different costume: anything that must not happen has to be made structurally impossible, or at least structurally hard — filtered at retrieval, blocked in the tool layer, absent from the context entirely. The prompt is the weakest layer to put a rule in, because it's the one layer the model can talk itself around. What survives is what the system enforces, not what the text requests. Your retrieval-layer framing is where the rule actually becomes a fact.
Exactly. I think the key distinction is that the prompt can guide behavior, but it can't undo a disclosure that already happened. Once restricted data is in context, you're relying on the model to respect a boundary the system itself failed to enforce. That’s a pretty fragile place to put security.
Exactly — and that asymmetry is why I moved the boundary out of the prompt entirely. Context is copyable and permanent for the session; an instruction is neither. The version that survived contact for me: the sensitive material never enters the context in the first place — the agent works with a reference, a handle, and only the step that legally needs the value ever resolves it, outside the conversation. Prompts are for judgement calls. Walls should be architecture.
I like that distinction between giving the agent the actual data and giving it a reference to the data. It takes the idea one step further: even if the agent gets manipulated, there's nothing sensitive in its context to expose. Keeping the resolution outside the conversation also gives you a much cleaner boundary to reason about. Definitely a stronger design.
'Even if the agent gets manipulated, there's nothing sensitive in its context to expose' — that's the formulation I'd put on the wall. Most security thinking asks how to make the agent resist manipulation. This asks how to make manipulation unrewarding, which is a much stronger position because it doesn't depend on the model behaving. Design for the compromise, not against it. The boundary you can reason about cleanly is the one that holds on the bad day, not the polite one.
"Design for the compromise, not against it" is a really good way to put it. We spend so much time trying to make models more robust against manipulation, but there's always going to be another way to influence their behavior. Removing access to the sensitive data in the first place changes the whole equation. Even a compromised agent shouldn't automatically mean compromised data.
Analyst focused on AI security
Adversarial evaluation of local LLMs, censorship and rejection rate, prompt injection testing, log analysis and defensive automation. Visit: sammideblas.com
Filtering after retrieval is the mistake we keep repeating. In our RAG pilot the vector query itself was scoped by a tenant tag in the metadata, so the model never saw a chunk the user was not allowed to read. Per permission indexes cost more work than one prompt line but they survive a paraphrased follow-up.
Exactly. A paraphrased follow-up is a good example of why prompt-level restrictions are so fragile. Scoping the query itself means the boundary holds regardless of how the user phrases the question or how the model interprets the instructions. The extra work upfront is definitely worth it when security is involved.
I test what ships and publish what I find, failures included.
Kubernetes, AWS and supply chain security.
Docker Captain, CNCF Ambassador, IBM Champion. 1.5M+ Docker Hub pulls.
Query time, in the tool rather than the retriever. Search is a tool on the MCP server that ranks only documents both the agent and the person it acts for may read, so rights are checked on every call and never copied into an index. The agent's own grant also caps the follow-up searches Gautam raised.
The audience sits in each document's front matter, which moves the risk rather than removing it. Tag a finance file audience: all and the control is wrong for everyone at once, so a static check refuses any audience nobody is granted.
Your per-role test is my gate. It reads the server's audit, never the answer. One run across six models: 954 of 954 returned documents within the person's rights.
That's a really interesting approach. I especially like that the permission check happens on every tool call rather than relying on a potentially stale copy in the index. And the static check for audience metadata is a nice extra safeguard. 954/954 is a solid result. Thanks for sharing the ADR, definitely worth a read.
AI CoolTalks is a community to explore AI tools, practical prompts, tutorials, and emerging AI trends. Discover ideas and discuss real-world applications of artificial intelligence.
Really good point. If restricted data reaches the model’s context, the permission check is already too late. Enforcing ACLs at the retrieval layer makes much more sense, especially when permissions can change over time. The point about testing retrieved chunks instead of only the final answer is particularly useful.
Thanks! The retrieved chunks are really the interesting part to test. Otherwise you can have two users getting seemingly identical answers while one of them was exposed to data they shouldn't even have had in context. That's the kind of issue that can easily go unnoticed in production.
One place I'd extend the role test: tool calls. In an agent style RAG setup the model often runs its own follow up searches, and those calls can go out with the service's identity instead of the user's, so the first retrieval is filtered and the second one isn't. Logging chunk IDs per role catches it only if the log covers every retrieval in the turn, not just the first.
In the setups you've reviewed, does the user's identity usually flow into the tool layer, or does it stop at the first query?
That's a great point. The tool layer is an easy place to accidentally lose the user's identity, especially when agents make follow-up calls through a shared service account. I'd treat every retrieval during the entire turn as part of the same authorization boundary. Otherwise the first query can be perfectly safe while the second one leaks everything.
Building AI systems and fixing AI-generated apps before they collapse in production.
Working with agentic AI, RAG pipelines, AI orchestration, and startup MVP scaling at IT Path Solutions.
One additional edge case worth addressing is permission inheritance across derived data. Chunk-level ACLs are important, but summaries, cached retrieval results, embeddings generated from restricted content, and conversation memory can all become secondary disclosure paths if they don't preserve the source permissions.
At IT Path Solutions, this is an important consideration when building production RAG systems. I'd treat authorization as part of the entire data lifecycle, not just the vector query: resolve current access at retrieval time, propagate the most restrictive ACL to derived artifacts, and invalidate cached results when permissions change.
The strongest test is exactly the one mentioned here: inspect the chunk IDs entering the model's context for each user role. A safe-looking final answer doesn't prove that restricted information was never exposed to the model.
Absolutely. The derived data part is especially easy to overlook because once content gets summarized, embedded or cached, people sometimes start treating it as a separate artifact with its own permissions. But the sensitivity doesn't disappear just because the representation changed. I like thinking about authorization across the entire data lifecycle rather than treating the vector query as the only enforcement point. That's where a lot of subtle leaks can happen.
A technologist working in automation. Here I write up what I actually learn building tools, engineering AI agents, and keeping systems running, along with the workflows and habits that stick.
Caching derived artifacts is where Failure 2 and Failure 3 collide in practice. Teams often push ACL filters into the vector query and still keep a separate semantic cache keyed only by prompt embedding or tenant ID, so a user whose group access dropped yesterday still hits Tuesday's cached answer. Keying the cache by sorted permission principals, or bypassing shared caches whenever restricted collections match, closes that path alongside query-time retrieval filtering.
Absolutely. Caching is another layer that can silently undo all the work done at retrieval time. Even with correct ACL filtering, a cached answer generated under an older permission set can still leak information. I'd also be careful with invalidation when group membership changes, because getting the cache key right doesn't help if stale entries remain accessible.
I write when an idea won't leave me alone 🧠 Building AI agents and the tools to build AI agents. Love connecting AI with other fields and yapping about all of them.
Query-time resolution for me, because any sync lag is a window where someone who just lost access still retrieves the chunk. Your top-k point is what I hadn't considered. A restricted folder full of near matches pushing every allowed result out means people who did nothing wrong quietly get worse answers, and nothing in the logs looks like a permissions problem.
Exactly, and that's the tricky part. We tend to think of authorization failures only as data leaks, but they can also affect availability and answer quality. A user getting incomplete results because inaccessible documents dominate the top-k is still a retrieval problem caused by permissions. Good reason to enforce filtering before ranking.
We're a place where coders share, stay up-to-date and grow their careers.
Thanks for this post
Thanks Ben, appreciate you taking the time to read it!
'Don't reveal it' is an instruction, and an instruction is a request, not a control. I run agents all day and this is the same lesson in a different costume: anything that must not happen has to be made structurally impossible, or at least structurally hard — filtered at retrieval, blocked in the tool layer, absent from the context entirely. The prompt is the weakest layer to put a rule in, because it's the one layer the model can talk itself around. What survives is what the system enforces, not what the text requests. Your retrieval-layer framing is where the rule actually becomes a fact.
Exactly. I think the key distinction is that the prompt can guide behavior, but it can't undo a disclosure that already happened. Once restricted data is in context, you're relying on the model to respect a boundary the system itself failed to enforce. That’s a pretty fragile place to put security.
Exactly — and that asymmetry is why I moved the boundary out of the prompt entirely. Context is copyable and permanent for the session; an instruction is neither. The version that survived contact for me: the sensitive material never enters the context in the first place — the agent works with a reference, a handle, and only the step that legally needs the value ever resolves it, outside the conversation. Prompts are for judgement calls. Walls should be architecture.
I like that distinction between giving the agent the actual data and giving it a reference to the data. It takes the idea one step further: even if the agent gets manipulated, there's nothing sensitive in its context to expose. Keeping the resolution outside the conversation also gives you a much cleaner boundary to reason about. Definitely a stronger design.
'Even if the agent gets manipulated, there's nothing sensitive in its context to expose' — that's the formulation I'd put on the wall. Most security thinking asks how to make the agent resist manipulation. This asks how to make manipulation unrewarding, which is a much stronger position because it doesn't depend on the model behaving. Design for the compromise, not against it. The boundary you can reason about cleanly is the one that holds on the bad day, not the polite one.
"Design for the compromise, not against it" is a really good way to put it. We spend so much time trying to make models more robust against manipulation, but there's always going to be another way to influence their behavior. Removing access to the sensitive data in the first place changes the whole equation. Even a compromised agent shouldn't automatically mean compromised data.
Filtering after retrieval is the mistake we keep repeating. In our RAG pilot the vector query itself was scoped by a tenant tag in the metadata, so the model never saw a chunk the user was not allowed to read. Per permission indexes cost more work than one prompt line but they survive a paraphrased follow-up.
Exactly. A paraphrased follow-up is a good example of why prompt-level restrictions are so fragile. Scoping the query itself means the boundary holds regardless of how the user phrases the question or how the model interprets the instructions. The extra work upfront is definitely worth it when security is involved.
Query time, in the tool rather than the retriever. Search is a tool on the MCP server that ranks only documents both the agent and the person it acts for may read, so rights are checked on every call and never copied into an index. The agent's own grant also caps the follow-up searches Gautam raised.
The audience sits in each document's front matter, which moves the risk rather than removing it. Tag a finance file
audience: alland the control is wrong for everyone at once, so a static check refuses any audience nobody is granted.Your per-role test is my gate. It reads the server's audit, never the answer. One run across six models: 954 of 954 returned documents within the person's rights.
ADR 21
That's a really interesting approach. I especially like that the permission check happens on every tool call rather than relying on a potentially stale copy in the index. And the static check for audience metadata is a nice extra safeguard. 954/954 is a solid result. Thanks for sharing the ADR, definitely worth a read.
Really good point. If restricted data reaches the model’s context, the permission check is already too late. Enforcing ACLs at the retrieval layer makes much more sense, especially when permissions can change over time. The point about testing retrieved chunks instead of only the final answer is particularly useful.
Thanks! The retrieved chunks are really the interesting part to test. Otherwise you can have two users getting seemingly identical answers while one of them was exposed to data they shouldn't even have had in context. That's the kind of issue that can easily go unnoticed in production.
One place I'd extend the role test: tool calls. In an agent style RAG setup the model often runs its own follow up searches, and those calls can go out with the service's identity instead of the user's, so the first retrieval is filtered and the second one isn't. Logging chunk IDs per role catches it only if the log covers every retrieval in the turn, not just the first.
In the setups you've reviewed, does the user's identity usually flow into the tool layer, or does it stop at the first query?
That's a great point. The tool layer is an easy place to accidentally lose the user's identity, especially when agents make follow-up calls through a shared service account. I'd treat every retrieval during the entire turn as part of the same authorization boundary. Otherwise the first query can be perfectly safe while the second one leaks everything.
One additional edge case worth addressing is permission inheritance across derived data. Chunk-level ACLs are important, but summaries, cached retrieval results, embeddings generated from restricted content, and conversation memory can all become secondary disclosure paths if they don't preserve the source permissions.
At IT Path Solutions, this is an important consideration when building production RAG systems. I'd treat authorization as part of the entire data lifecycle, not just the vector query: resolve current access at retrieval time, propagate the most restrictive ACL to derived artifacts, and invalidate cached results when permissions change.
The strongest test is exactly the one mentioned here: inspect the chunk IDs entering the model's context for each user role. A safe-looking final answer doesn't prove that restricted information was never exposed to the model.
Absolutely. The derived data part is especially easy to overlook because once content gets summarized, embedded or cached, people sometimes start treating it as a separate artifact with its own permissions. But the sensitivity doesn't disappear just because the representation changed. I like thinking about authorization across the entire data lifecycle rather than treating the vector query as the only enforcement point. That's where a lot of subtle leaks can happen.
Caching derived artifacts is where Failure 2 and Failure 3 collide in practice. Teams often push ACL filters into the vector query and still keep a separate semantic cache keyed only by prompt embedding or tenant ID, so a user whose group access dropped yesterday still hits Tuesday's cached answer. Keying the cache by sorted permission principals, or bypassing shared caches whenever restricted collections match, closes that path alongside query-time retrieval filtering.
Absolutely. Caching is another layer that can silently undo all the work done at retrieval time. Even with correct ACL filtering, a cached answer generated under an older permission set can still leak information. I'd also be careful with invalidation when group membership changes, because getting the cache key right doesn't help if stale entries remain accessible.
Query-time resolution for me, because any sync lag is a window where someone who just lost access still retrieves the chunk. Your top-k point is what I hadn't considered. A restricted folder full of near matches pushing every allowed result out means people who did nothing wrong quietly get worse answers, and nothing in the logs looks like a permissions problem.
Exactly, and that's the tricky part. We tend to think of authorization failures only as data leaks, but they can also affect availability and answer quality. A user getting incomplete results because inaccessible documents dominate the top-k is still a retrieval problem caused by permissions. Good reason to enforce filtering before ranking.