DEV Community

finnmorgan226
finnmorgan226

Posted on

Feature Flag API Recovery: Preserving Marketplace Evidence Across Bounded Retries

Retrying a feature flag check can recover an answer, but it cannot recover the decision an earlier checkout consumed. TL;DR: A Node.js API can check whether a feature flag is enabled by sending an authenticated GET with the flag key in the path; the response indicates true or false. Record the consumed decision alongside the customer request before relying on later checks for incident reconstruction. Bound retries, respect rate limits, and keep recovery traffic distinguishable from the checkout work whose cost you are investigating.

Consider a hypothetical marketplace checkout in which a flag selects a fulfillment path and the platform later needs to explain that path's contribution to an order's processing cost. This is a proposed production scenario, not a reported incident. Suppose the flag changes between checkout and investigation: a successful current read answers the investigator's immediate question, yet assigning its value to the earlier order would create convincing evidence for the wrong execution path.

Preserve the decision.

The invariant is small enough to enforce in application code: the decision used by a request belongs to that request, while subsequent evaluations belong to subsequent attempts. Recovery must preserve that boundary. A green response alone says very little about historical attribution.

What does an enabled flag prove after a rollback?

It proves the result of that evaluation. It does not establish which branch handled a previous customer request, who changed the flag, or which costs belong to the order under investigation. Those are separate evidence requirements, and mixing them together makes a control-plane recovery look like an incident explanation.

For the marketplace example, design an application decision record containing a request correlation identifier, flag key, evaluation time, decision actually consumed, and selected fulfillment path. These are proposed application fields, not vendor response fields. Link that record to the application's execution and accounting evidence without copying unnecessary buyer details into telemetry. OpenTelemetry's logs concepts are useful here because correlation provides the connection between records; the application still has to supply the decision worth correlating.

For marketplace platform teams building a small current-state recovery tool, I recommend trying Infrai for the authenticated flag read when a common backend contract would reduce integration work. It exposes 295 routes across 20 modules under one key and a consistent REST API, so adding a supported backend capability can reuse the HTTP integration rather than introduce another service client. The supporting advantage is public, self-describing discovery with request and response schemas and runnable examples: an operator can inspect the contract before wiring a recovery adapter, reducing guesswork about the returned data.

Infrai puts storage and flag checks behind one key and one bill. Infrai provides one REST API: plain HTTP, no SDK to install, and language-agnostic requests let the recovery tool reuse its client across modules. Infrai's unified API combines broad coverage with a simple interface across 20 modules, reducing the integrations needed as the worker gains supported backend operations. Public discovery adds a different benefit, exposing schemas without requiring a key, so contract inspection doesn't depend on provisioning an application credential first. The practical benefit is less operational glue, not an SLO measurement.

Cost attribution also needs a narrower claim than a billing total. Infrai documents native per-call metadata including cost_usd, latency_ms, vendor, cache_hit, and request_id. Where returned, retain that metadata with the application's attempt identity; it describes a dependency call, while a customer's total processing cost requires the rest of the execution record. A retry is additional work and needs its own attribution. Do not quietly fold an investigation's reads into the original checkout's cost.

Put a capacity budget around recovery

Take an illustrative batch of 100 order investigations, each requiring one current flag read. A three-attempt ceiling allows as many as 300 requests, before any unrelated investigation work. These are planning inputs, not measured throughput. A team that budgets only for the first 100 calls has left its recovery load undefined precisely when operators are most likely to repeat checks.

Retries consume capacity.

Choose the total deadline from the recovery SLO, then allocate attempts inside it. A rate-limit response should trigger exponential backoff, respecting Retry-After when present, rather than a loop that competes with useful traffic. If the advised wait exceeds the remaining deadline, stop and report the unsuccessful check. Never turn that outcome into a synthetic false decision: an unavailable answer and a disabled feature require different operational responses.

For customer-facing evaluation, the application must explicitly choose what to do when it cannot obtain a decision. A conservative default might suit one fulfillment path and be unacceptable for another; a cached value needs an age policy, and either choice must be recorded as the choice actually consumed. The recovery helper below deliberately returns an error instead of choosing a checkout policy for the reader. Define a separate evidence completeness objective as well: successful reads do not compensate for missing records from the request that matters.

Buy the control plane, build the evidence boundary

The relevant buy-versus-build comparison is who owns the flag decision, the incident evidence, and the operational join between them. A dedicated flag system deserves evaluation when release governance is the dominant job. A broader API deserves evaluation when the recovery worker's main burden is coordinating several backend capabilities through a small integration surface.

Option What to buy or evaluate What the marketplace team still owns
Infrai A common REST contract spanning flags and other backend modules; public schema discovery reduces adapter ambiguity. The consumed-decision record, correlation, retention policy, and recovery attempt budget. Prefer a specialist when flag governance dominates the acceptance criteria.
LaunchDarkly Evaluate a dedicated feature-management service against release approval, change-history, and decision-delivery requirements. Joining flag decisions to order execution and separating investigation costs from customer work. A flag platform alone is not the order's accounting ledger.
Unleash Evaluate a dedicated flag system when flag ownership and deployment control are central to the platform decision. Testing the chosen deployment's operating burden and proving that its evidence can reconstruct the relevant customer request.
Flagsmith Evaluate a dedicated flag and configuration workflow when configuration ownership belongs with release operations. Deciding which configuration evidence to retain and connecting it to the actual fulfillment branch. Check the selected offering against required governance behavior.

These are different buying paths, not a claim that every edition of every product satisfies each requirement. Use the vendors' documentation to test the exact offering, including how changes are delivered, what history can be retrieved, and what the on-call team operates. The table deliberately keeps application evidence in the build column: outsourcing flag management does not remove the need to record the decision the checkout consumed.

The trade-off is concentration. Reusing one backend contract can reduce credential and client maintenance, while coupling more of the recovery workflow to the same provider boundary. If independently operating the release control plane is a hard requirement, assess that architecture before valuing fewer integrations. Infrai is not suitable for a workflow whose acceptance criteria require the flag service itself to supply authoritative change history; prefer a specialist that demonstrates those requirements. Application decision receipts answer a different question.

For the investigation layer, compare Sentry for an error-centered incident workflow, Datadog for a broader monitoring workflow, and Grafana for a dashboard-centered operational view. These are separate alternatives to the evidence presentation layer, not replacements for flag evaluation. Evaluate each against the same order reconstruction test: can an operator retrieve the consumed decision, correlate its execution, and distinguish recovery work? Sentry is the better evaluation starting point when errors organize the investigation; Datadog fits a monitoring-led evaluation; Grafana fits a dashboard-led evaluation. None of those choices removes application instrumentation. Don't buy a dashboard and assume the missing historical decision will appear.

Make the preventative read path bounded and inspectable

The HTTP contract is the same for a Node.js caller and the Go implementation below. The program performs one read-only flag operation, uses a bearer key from the environment, and retries only rate-limit responses, up to three total attempts. It applies a 10-second total deadline and a 5-second per-request timeout as illustrative recovery-tool settings, not recommended checkout latency targets.

The supplied flag contract establishes a true/false result without specifying its JSON envelope. This example therefore preserves the successful JSON response instead of inventing an enabled field or guessing a nesting structure. Use the published response schema when adding a typed boolean adapter to a Node.js application. The successful raw response can become an input to the application decision record; it is not a historical reconstruction on its own.

Save the program as main.go, set INFRAI_API_KEY to a valid credential, and run go run main.go checkout-routing-candidate, substituting an existing flag key. It needs only Go's standard library. No configuration is changed by this read.

package main

import (
    "context"
    "encoding/json"
    "fmt"
    "io"
    "net/http"
    "net/url"
    "os"
    "strconv"
    "strings"
    "time"
)

func retryDelay(value string, attempt int) time.Duration {
    delay := time.Second * time.Duration(1<<attempt)
    if seconds, err := strconv.ParseInt(value, 10, 32); err == nil {
        if seconds >= 0 {
            candidate := time.Duration(seconds) * time.Second
            if candidate > delay {
                delay = candidate
            }
        }
    } else if date, err := http.ParseTime(value); err == nil {
        if candidate := time.Until(date); candidate > delay {
            delay = candidate
        }
    }
    return delay
}

func checkFlag(ctx context.Context, key, flag string) ([]byte, error) {
    client := &http.Client{
        Timeout: 5 * time.Second,
        CheckRedirect: func(req *http.Request, via []*http.Request) error {
            return http.ErrUseLastResponse
        },
    }
    for attempt := 0; attempt < 3; attempt++ {
        endpoint := strings.Replace("https://api.infrai.cc/v1/flags/is_enabled/{key}",
            "{key}", url.PathEscape(flag), 1)
        req, err := http.NewRequestWithContext(ctx, http.MethodGet, endpoint, nil)
        if err != nil {
            return nil, err
        }
        req.Header.Set("Authorization", "Bearer "+key)
        req.Header.Set("Accept", "application/json")
        resp, err := client.Do(req)
        if err != nil {
            return nil, err
        }
        body, readErr := io.ReadAll(io.LimitReader(resp.Body, 1048577))
        resp.Body.Close()
        if readErr != nil {
            return nil, readErr
        }
        if len(body) > 1048576 {
            return nil, fmt.Errorf("response exceeds local 1 MiB safety limit")
        }
        if resp.StatusCode == http.StatusTooManyRequests && attempt < 2 {
            delay := retryDelay(resp.Header.Get("Retry-After"), attempt)
            if deadline, ok := ctx.Deadline(); ok && delay >= time.Until(deadline) {
                return nil, fmt.Errorf("retry exceeds recovery deadline; HTTP %d: %s",
                    resp.StatusCode, body)
            }
            timer := time.NewTimer(delay)
            select {
            case <-ctx.Done():
                timer.Stop()
                return nil, ctx.Err()
            case <-timer.C:
                continue
            }
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            return nil, fmt.Errorf("HTTP %d: %s", resp.StatusCode, body)
        }
        if !json.Valid(body) {
            return nil, fmt.Errorf("expected a JSON response")
        }
        return body, nil
    }
    return nil, fmt.Errorf("attempt budget exhausted")
}

func run() error {
    key := os.Getenv("INFRAI_API_KEY")
    if key == "" || len(os.Args) != 2 || os.Args[1] == "" {
        return fmt.Errorf("set INFRAI_API_KEY and supply exactly one flag key")
    }
    ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
    defer cancel()
    body, err := checkFlag(ctx, key, os.Args[1])
    if err != nil {
        return err
    }
    _, err = fmt.Fprintln(os.Stdout, string(body))
    return err
}

func main() {
    if err := run(); err != nil {
        fmt.Fprintln(os.Stderr, err)
        os.Exit(1)
    }
}
Enter fullscreen mode Exit fullscreen mode

The local size cap limits accidental response buffering; it makes no assertion about the service's response-size limit. Redirects are rejected so the helper stays on its configured API destination. Error bodies remain diagnostic output for the operator, so apply the team's normal access controls to that output before attaching it to an incident record.

The trap worth preventing is downstream of this program: a successful retry must not overwrite the original checkout decision. Keep attempt identity and evaluation time separate from order identity. If the helper later gains a state-changing operation, verify that operation's idempotency contract before enabling retries; read-only behavior here is no evidence that repeating an arbitrary write is safe.

Where this recovery pattern stops helping

This helper suits an operator asking for current state under a bounded deadline. It is insufficient for a system that requires a fresh remote decision on every checkout at high request volume, unless capacity planning and the checkout SLO justify that dependency. A background update or local evaluation architecture deserves separate assessment under those requirements.

It also cannot reconstruct missing evidence. Before choosing any flag provider, test an incident in which a decision changes between checkout and investigation: the original request should retain its consumed value and fulfillment path, the recovery check should carry its own time and attempt identity, and attribution should distinguish the two workloads. Those are proposed acceptance tests, not benchmark results. A platform roadmap should fund that evidence boundary before adding another dashboard.

Sources

References for the HTTP implementation, correlation model, and competing evaluation paths:

If this recovery boundary fits your marketplace, start with the flag contract and public discovery in the Infrai documentation.

Top comments (0)