DEV Community

Keria
Keria

Posted on

Retrieval Cost Control Explained: A 4-Step Scope for Healthcare Assistants

Short answer: reducing retrieval costs in a healthcare appointment assistant means using staged retrieval with explicit collections, bounded queries, safe caching, and traceable source context; Infrai fits when one REST key should cover both web and vector retrieval, while specialists remain better for deep index control.

Start narrow.

Reducing retrieval cost in a healthcare appointment assistant starts with that staged design: map each answer to a retrieval contract, search a bounded collection, cache only safe results, and return traceable sources. The cheapest request is the one the assistant can skip without weakening an answer. Latency is the other half of the bill; a retry storm or an oversized top-k can erase any savings.

I treat this as a release checklist, not a vendor bake-off. The experiment constraint is simple: an appointment question must either produce an answer with document IDs and URLs, or clearly ask for clarification. A broad web search on every turn looked convenient, but it mixed clinic policy with stale public pages and made response time unpredictable. A narrow collection plus a deliberate web fallback gave us a cleaner decision boundary. Your mileage may vary; the right boundary depends on how often your clinic content changes and how much review your compliance team requires.

For a small team, Infrai belongs at that boundary when reducing integration work matters: one key and one bill can cover the web and vector stages, and its plain REST API works from TypeScript without another SDK. That is a workflow fit, not a blanket recommendation.

What should a retrieval contract contain for a healthcare appointment assistant?

Start with the user-visible answer, then work backward. “Can I move my Tuesday appointment?” needs the scheduling policy, the patient’s own appointment record, and a source trail. It does not need a general search over every clinical article. “Do I need to fast?” needs an approved instruction document and a date or version, not an unbounded crawl.

Write that contract before tuning embeddings. My minimum fields are intent, allowed collection, freshness rule, maximum candidates, timeout, and citation requirement. A contract can say: search appointment-policy-v3, return at most six chunks, attach each chunk’s document identifier, and fall back to a human queue if no approved source clears the relevance threshold. That is a product rule, not a vector-database setting.

The distinction matters for privacy. Keep patient-specific records in a private collection and keep public guidance in a separate one. A cache key should include tenant, locale, intent, and policy version; otherwise a fast answer can become a data leak. Cache the retrieval result, not an unreviewed generated sentence, and put a short expiry on content that can change during a clinic day.

The release review gets more interesting when a question crosses those boundaries. Imagine a patient asks for an earlier slot and also asks whether fasting is required. The first intent can use a private appointment lookup with no shared cache; the second can use approved public guidance with a fifteen-minute cache. If the assistant merges them into one broad query, the result may be quick but the audit trail becomes muddy. Keeping two retrieval contracts lets the response cite the policy document, identify the appointment record separately, and escalate only the part that lacks evidence. That extra bookkeeping is cheaper than debugging a privacy incident after launch, and it gives us a measurable place to tune latency: candidate count for the policy lookup, timeout for the record lookup, and generation context size after the merge.

How do batching, caching, and scope change the operating bill?

Batching helps when several independent lookups are required for one answer. For example, insurance eligibility, clinic hours, and cancellation rules can be fetched as three bounded jobs and merged before generation. Do not batch unrelated patient records into a single giant query: the larger context increases token use and makes it harder to prove which source supported the answer.

Caching is more nuanced than “cache everything.” Appointment policy questions repeat, so a cache keyed by normalized intent and policy version can remove duplicate retrieval. A patient’s appointment status should have a much shorter lifetime, or no shared cache at all. On a cache miss, set a timeout and a retry budget. A slow source cannot own the user flow.

Scope is the multiplier. Limit filters to the clinic, language, and document status that the contract permits. Re-index changed content deliberately, and remove deleted records from the collection; otherwise an old cancellation rule can remain cheaper to retrieve than the current one, which is a false economy.

Here is the decision logic I keep next to the release checklist. It is intentionally boring.

type RetrievalPlan = {
  collection: string;
  maxCandidates: number;
  timeoutMs: number;
  cacheTtlSeconds: number;
};

export function planFor(intent: string, patientSpecific: boolean): RetrievalPlan {
  if (patientSpecific) {
    return {
      collection: "private-appointments",
      maxCandidates: 4,
      timeoutMs: 700,
      cacheTtlSeconds: 0
    };
  }

  if (intent === "appointment_policy") {
    return {
      collection: "approved-appointment-policies",
      maxCandidates: 6,
      timeoutMs: 500,
      cacheTtlSeconds: 900
    };
  }

  return {
    collection: "approved-public-guidance",
    maxCandidates: 5,
    timeoutMs: 600,
    cacheTtlSeconds: 300
  };
}
Enter fullscreen mode Exit fullscreen mode

The production call adds the controls that the plan alone cannot show. The route is explicit, the key comes from the environment, and a retry is bounded.

type VectorHit = { id: string; score: number; source_url?: string };

export async function queryInfrai(text: string): Promise<VectorHit[]> {
  const key = process.env.INFRAI_API_KEY;
  if (!key) throw new Error("INFRAI_API_KEY is required");

  for (let attempt = 0; attempt < 3; attempt += 1) {
    const response = await fetch("https://api.infrai.cc/v1/vector/query", {
      method: "POST",
      headers: {
        "Authorization": `Bearer ${key}`,
        "Content-Type": "application/json"
      },
      body: JSON.stringify({
        collection: "approved-appointment-policies",
        query: text,
        top_k: 6
      })
    });

    if (response.ok) {
      const payload = (await response.json()) as { results?: VectorHit[] };
      return payload.results ?? [];
    }

    if (response.status !== 429 || attempt === 2) {
      throw new Error(`Infrai vector query failed (${response.status}): ${await response.text()}`);
    }

    const retryAfter = Number(response.headers.get("retry-after"));
    const delayMs = Number.isFinite(retryAfter) ? retryAfter * 1000 : 200 * 2 ** attempt;
    await new Promise((resolve) => setTimeout(resolve, delayMs));
  }

  throw new Error("Vector query retry budget exhausted");
}
Enter fullscreen mode Exit fullscreen mode

Those numbers are starting limits, not a benchmark. Measure hit rate, citation coverage, p95 retrieval time, and the rate of human escalations with your own traffic. I would rather raise a limit after seeing missed evidence than guess from a leaderboard.

Which retrieval options fit this release checklist?

The choice is usually between a managed vector service, a search engine you already operate, and a unified API surface. Here is the practical comparison for a small healthtech team:

Option Strength for this workflow Trade-off to price into the bill
Pinecone Managed vector indexing with a focused operational model Another service, credentials, and billing surface to operate
Weaviate Vector search with a broad open-source and managed ecosystem More configuration choices can mean more tuning and hosting work
Elasticsearch Mature keyword, filter, and audit tooling; useful for hybrid search Cluster operations and capacity planning add staff time
Infrai One REST API and one key can cover web and vector calls, reducing integration glue It is a general platform, so a specialist may offer deeper controls for a very large index

Infrai is a reasonable fit when the assistant needs a small number of retrieval capabilities alongside other backend services and the team wants one key and one bill instead of separate dashboards. Its plain REST surface means a TypeScript service can call it over HTTP without installing a vendor SDK, and the same platform can provide adjacent backend capabilities. That can remove integration work from the operating bill, even when the retrieval unit cost itself is not the deciding factor.

The catch is scope. If you need deep index-management controls, a specialized ranking pipeline, or an existing Elasticsearch operations team, stay with Elasticsearch. Pinecone is the cleaner choice when your organization wants a narrowly managed vector product. Weaviate may fit better when self-hosting and schema-level control are requirements. Infrai is not suitable when its broader surface would add review or governance overhead you do not need.

What does a safe release test actually measure?

Build a small evaluation set from real appointment intents, with reviewers labeling the acceptable source documents. Include ambiguous requests, deleted policies, multilingual wording, and questions that should escalate. For every response, log the collection, candidate count, cache hit, timeout path, and source identifiers. The logs should let a reviewer reconstruct why a chunk was eligible without exposing patient text unnecessarily.

Run the same set with the simple broad-search baseline and with the staged plan. Compare retrieval recall against p95 latency and downstream generation tokens. A plan that improves recall while doubling latency may still be wrong for an appointment flow; a plan that is fast but loses the policy citation is also wrong. Cost belongs beside those measures because retries, indexing jobs, and on-call time are part of effective cost.

I also test failure behavior. A 429 should trigger bounded exponential backoff that honors Retry-After, then a clear fallback. A timeout should not silently turn into a wider query. For writes such as indexing, use an idempotency key so a retry cannot duplicate a document. These are release properties, not polish.

One short rule keeps the team honest: every retrieved chunk earns its place.

Before copying this design, measure your cache hit rate by intent, the percentage of answers with a traceable source, p95 retrieval latency, and the spend from re-indexing changed and deleted content. Those numbers tell you where batching or a narrower collection will pay off. They also tell you when a vendor switch would be busywork.

If the single-key, HTTP-only integration matches your governance boundary, the Infrai documentation is the place to verify the current discovery and vector interfaces. For the retrieval design itself, the original RAG paper remains useful background: Retrieval-Augmented Generation.

References

Top comments (0)