Short answer: build legal contract review retrieval around a replayable evidence ledger, then retain three different layers for three different periods: source revisions for authority, retrieval receipts for audit and replay, and replaceable index artifacts only for fast service restoration.
That choice puts the expensive question in the right place. The bill is not merely the vector index. It is the sum of source retention, parsed text, embeddings, lexical and vector replicas, reranking work, query receipts, and the engineering time required to prove why a clause appeared in a review. In a contract workspace, I would first measure stored bytes by artifact class and retrieval compute by stage. Don't optimize a per-query detail while duplicate index generations dominate storage.
The practical rule is blunt: retain what proves the answer, regenerate what only accelerates it.
What should a legal contract review workspace retain for retrieval failure recovery?
Start with a cost ledger, not a diagram. For each tenant and matter, record source bytes, normalized-text bytes, chunk count, embedding bytes, index bytes per generation, replicas, receipt bytes, and reranking calls. No universal multiplier belongs here; document length, chunk policy, embedding width, metadata, and index implementation all change it. Your mileage may vary, so measure those terms on a representative matter before selecting a recovery design.
This small model makes the dominant term visible without pretending that an estimate is a benchmark:
from dataclasses import dataclass
@dataclass(frozen=True)
class RetrievalFootprint:
source_bytes: int
normalized_bytes: int
embedding_bytes: int
index_bytes: int
index_generations: int
replicas: int
receipt_bytes: int
def retained_bytes(self) -> int:
searchable = (
self.embedding_bytes + self.index_bytes
) * self.index_generations * self.replicas
return (
self.source_bytes
+ self.normalized_bytes
+ searchable
+ self.receipt_bytes
)
def largest_term(self) -> tuple[str, int]:
terms = {
"source": self.source_bytes,
"normalized": self.normalized_bytes,
"searchable_generations": (
self.embedding_bytes + self.index_bytes
) * self.index_generations * self.replicas,
"receipts": self.receipt_bytes,
}
return max(terms.items(), key=lambda item: item[1])
Run that calculation with observed inventory values. If searchable generations are largest, the meaningful change is to reduce retained generations or replicas after a verified cutover. If source revisions dominate, deleting index generations won't move the total much. If reranking dominates the operating bill rather than stored bytes, narrow the candidate set only after checking that the relevant clause still reaches the reranker. Cost follows workload shape — and relevance has to veto a cheap but incomplete shortcut.
The three retention layers have separate jobs. Layer one is the authoritative contract revision plus stable document identity and access-control metadata. Layer two is a compact retrieval receipt: query identity, source revision set, filter policy version, chunk identifiers, stage scores, reranker version, and the final ordering. Layer three is the serving material: normalized chunks, embeddings, lexical structures, and active or previous index generations. Mixing these lifetimes is how teams end up keeping every index forever because one audit field was trapped inside it.
Make the receipt the recovery boundary
A retrieval receipt should answer two questions: what evidence was eligible, and what transformations produced the order shown to the reviewer? It does not need to copy the full contract text when stable chunk identifiers resolve back to an immutable source revision. It does need enough version information to distinguish a changed corpus from a changed ranking policy.
RAG joins retrieval with generation; the original RAG paper describes retrieved passages as external knowledge used by the generator. For contract review, preserve that boundary even if the user interface presents one fluent answer. A generated explanation is not a substitute for the eligible source set, and a final rank alone cannot explain which candidates were filtered out before reranking.
Use explicit states. Quiet ambiguity is worse than a visible degraded mode.
from dataclasses import dataclass
from enum import Enum
class RetrievalMode(str, Enum):
FULL = "full"
BASELINE = "baseline"
SOURCE_ONLY = "source_only"
@dataclass(frozen=True)
class RecoveryDecision:
mode: RetrievalMode
revision_id: str
policy_version: str
candidate_ids: tuple[str, ...]
reason: str
def choose_mode(
*,
active_index_matches_revision: bool,
baseline_index_matches_revision: bool,
revision_id: str,
policy_version: str,
candidates: tuple[str, ...],
) -> RecoveryDecision:
if active_index_matches_revision:
return RecoveryDecision(
RetrievalMode.FULL,
revision_id,
policy_version,
candidates,
"active generation verified",
)
if baseline_index_matches_revision:
return RecoveryDecision(
RetrievalMode.BASELINE,
revision_id,
policy_version,
candidates,
"revision-matched baseline selected",
)
return RecoveryDecision(
RetrievalMode.SOURCE_ONLY,
revision_id,
policy_version,
(),
"search results withheld pending revision-matched rebuild",
)
The important detail is the revision match. A fast index for yesterday's contract is not a valid recovery target for today's amendment. In SOURCE_ONLY, the workspace can still expose the authoritative document and navigation, but it should not label an unverified result list as complete. That's a compliance-aware failure mode: the reviewer sees less automation, not invented confidence.
Keep authorization in the same eligibility path during normal and recovery modes. A fallback that skips matter-level or document-level filtering is not recovery; it is a different security policy. The receipt should store the policy version and eligible identifiers, while sensitive query text and copied clauses should follow the workspace's own retention and access rules. I'm not sure a single receipt lifetime can serve every jurisdiction or matter type. Counsel and records owners have to decide that policy; engineering can make it enforceable and observable.
Recovery should spend latency deliberately
Retrieval quality and latency pull against each other most sharply during rebuilds. A lexical baseline over the correct source revision may return quickly but miss paraphrased obligations. A broader hybrid candidate stage may improve coverage while adding work. A reranker can reorder only what it receives, so starving its candidate pool is an irreversible quality loss inside that request.
Treat recovery as a state machine with admission rules rather than an automatic retry loop. First, pin every request to a source-revision manifest. Next, select the newest verified index generation that exactly matches that manifest and authorization policy. If the richer candidate path is unavailable within its latency budget, use a revision-matched baseline and mark the receipt accordingly. If no verified generation matches, withhold derived search results and initiate rebuild from retained source artifacts. The reviewer can still inspect the contract directly.
Slow is sometimes correct.
A useful service objective separates response time from completeness. Track latency for each mode, but also track the fraction of requests served by baseline mode, the age of the matched revision, candidate-set size before reranking, overlap between old and new top results on a fixed evaluation set, and the number of requests withheld for lack of a matching generation. A single latency percentile hides the exact edge case that matters: a quick answer built from stale evidence.
Retries deserve the same discipline. Retry idempotent build steps against a generation identifier, cap attempts, and preserve the last verified generation until the replacement passes checks. Do not merge partial output into the active generation. A retry that silently mixes chunk versions can look healthy to infrastructure monitoring while changing the legal evidence set underneath the reviewer.
Test relevance before promoting an index generation
Promotion needs both integrity tests and relevance tests. Integrity checks confirm that every indexed chunk belongs to the intended source revision, every identifier resolves, access filters match the policy version, and expected document counts reconcile with the manifest. Relevance checks use a versioned set of contract-review questions with adjudicated clauses, including amendments, defined terms, negation, dates, tables, and cross-references.
I would keep the evaluation set small enough to run on every candidate generation and broad enough to contain the ugly cases. "Termination" queries are easy. The harder cases ask whether a later amendment narrows an earlier termination right, or whether a defined term changes the meaning of a clause several pages away. Those examples expose chunk boundaries and candidate starvation without requiring a production incident story.
Compare a candidate generation with the current verified generation under the same source manifest. Inspect recall at the candidate boundary before judging reranked order; otherwise a ranking change may be blamed for a passage that retrieval never supplied. Then sample the actual receipts for unexpected filter changes and score distributions. Promotion is allowed only after integrity invariants pass and the agreed relevance threshold is met. The precise threshold is a local risk decision, not a number that can be borrowed from another corpus.
Deployment follows a simple sequence: build under a new generation ID, validate it, shadow representative queries, atomically switch the manifest pointer, and retain the previous verified generation for a bounded rollback window. Requests already in flight stay pinned to their original manifest. This costs temporary duplicate storage, but it avoids half-old, half-new result sets and makes rollback a pointer change instead of an emergency rebuild.
The catch is that this architecture is not suitable when the organization cannot preserve immutable source revisions or define stable chunk identities. Fix that provenance layer first. For a tiny workspace where reviewers can tolerate search downtime and manually reopen the source contract, a single rebuildable index plus verified backups may be the better trade-off; the receipt machinery can cost more operational attention than it returns. For high-volume review with strict interactive latency, retain more than one verified serving generation or provision warm capacity, accepting the additional storage bill.
Stop retaining replaceable acceleration artifacts
After the rollback window closes and receipts resolve correctly against retained source revisions, delete superseded embeddings and index generations. Keep the authoritative revision, the parser and chunk-policy version, reproducible build configuration, integrity manifests, and the receipts required by the approved audit policy. Also aggregate operational metrics before deleting request-level detail, so trend analysis does not require indefinite retention of sensitive queries.
This is the change that moves the dominant storage term when replicated generations are the largest item: stop treating every serving artifact as evidence. The trade-off is real. A later investigation can replay eligibility and ranking from the receipt and retained source, but exact reconstruction may require rebuilding an old parser, embedding, or reranker version. If exact bit-for-bit reproduction is mandatory, retain the corresponding artifacts and accept their storage, access-control, and lifecycle burden. If decision traceability is the requirement, a versioned receipt plus authoritative source may be enough — but records owners, not the search team alone, must make that call.
Recovery is credible when the workspace can say which revision it searched, which policy it applied, which mode it used, and why the reviewer saw those clauses. Everything else is acceleration.
Top comments (0)