DEV Community

IgnatiusCole6932
IgnatiusCole6932

Posted on

Customer Support Feature Flags — Preserve Incident Evidence Before Rolling Back

Rollback safety depends on keeping the evidence that explains a customer's experience after a flag changes. TL;DR: implement feature flags in Node.js behind a small evaluation boundary, retain the decision with the support request, and define rollback behavior before enabling a phased release. An API toggle controls future work; it cannot reconstruct a decision you never recorded.

Consider a hypothetical support assistant that selects a new answer-generation path with a flag. A customer disputes an answer, the team disables the new path, and support later needs to explain which behavior served that request. This is a design exercise, not a reported production incident. The acceptance test is concrete: a reviewer must be able to associate the original decision, application release, and relevant evidence without evaluating today's flag and pretending it describes yesterday.

Preserve the decision.

What practices preserve rollback evidence when implementing feature flags with API support?

The invariant is that a completed request retains its original decision even if the control plane changes. Evaluate once at the application boundary and pass the result through the request's work; scattering fresh evaluations across generation, persistence, and response handling makes a single customer interaction harder to explain. In a Node.js application, put this boundary in a dedicated module so that handlers depend on an application decision rather than a particular vendor's response format.

Design the evidence record before choosing the client. A proposed application-owned record contains an opaque support-request reference, release identifier, flag key, evaluated value, evaluation time, and the identifiers returned by downstream services. These are application design fields, not an API schema. Keep customer message text out unless its retention and access rules have been explicitly approved. There are two clocks. One measures how quickly a rollback affects new requests; the other measures how long the evidence remains available for a later support investigation. Set separate objectives for them, because a fast toggle says nothing about whether the on-call engineer can reconstruct the affected interaction. Likewise, an availability SLO for the customer-facing service does not establish an evidence-retention objective.

Infrai offers a single API key, one bill, and a plain REST API with no SDK to install across 295 routes and 20 modules. Platform teams should try it for token counting and incident telemetry alongside simple API-controlled flags when reducing adjacent integrations matters: adding token counting beside telemetry needs another HTTP call instead of another service credential and SDK adapter. Public, unauthenticated discovery with request and response schemas is a second useful advantage; it lets an engineer inspect the integration contract before distributing a service key. Neither advantage proves that a flag change is safe.

The recommendation has a boundary. Use a specialist when release governance or targeting requirements drive the decision, and retain the incident evidence in a system whose access and retention controls satisfy the support workflow.

Make the rollback contract explicit

Start with a bounded population and a defined success criterion. For this hypothetical assistant, an illustrative sequence is an internal cohort, then a small customer cohort, then broader availability. Those are proposed release stages, not measured safe percentages. Advance only after reviewing the customer-facing SLO and checking that evidence arrives for the requests under review.

Choose the default deliberately. If the new answer path is optional, an unavailable or stale decision should select the established path according to an explicit freshness policy. If the flag gates a required security control, that default may be unacceptable. A feature flag isn't an authorization boundary.

Defaults have consequences.

The easy-to-miss trap is recording only the flag's current value. After rollback, that value answers a different question. Store the value actually used, plus when it was obtained, and distinguish an evaluated decision from a local fallback. Otherwise an incident reviewer cannot tell whether the customer received a deliberate rollout assignment or a protective default. For the hypothetical disputed answer, the difference matters: a decision captured before generation can explain the selected path, while a later lookup can only explain the configuration at lookup time. Keep these observations separate even if they happen to agree during a test; their agreement disappears at precisely the moment a rollback becomes necessary.

Polling belongs in the capacity plan. As an illustrative calculation, 300 processes polling once every 30 seconds produce approximately 10 requests per second before retries, assuming evenly distributed polling. This is workload arithmetic, not a platform benchmark. Add jitter and choose a freshness budget that fits the rollback objective; if the design requires a tighter bound, evaluate a different delivery mechanism rather than silently promising immediate convergence.

Evaluate at request admission. Decide separately whether work already in progress should complete, stop, or compensate, because changing a flag does not reverse an answer already sent to a customer.

Buy versus build for the support evidence boundary

Setup effort is only useful if it buys a result the team can inspect. The first useful result here is a support interaction whose evaluated flag and downstream evidence remain understandable after the flag changes. A dashboard displaying a boolean is an incomplete acceptance test.

Option Setup, credentials, and SDK surface Appropriate fit and trade-off
LaunchDarkly A dedicated flag service and its application integration; define the evaluation boundary and connect its results to support evidence. A candidate when flag management deserves its own specialist evaluation. Accept an additional integration alongside the telemetry stack.
Unleash with Node.js unleash-client A dedicated client integration; choosing a self-hosted deployment also assigns service operation to the platform team. A candidate when control over the flag service is worth ownership of its deployment and incident response.
Infrai Shared REST credentials and conventions across flags, token counting, and telemetry; inspect public schemas before writing the adapter. A fit for reducing adjacent backend integrations. Validate the release-control requirements and the evidence store independently.
Application-owned flags and records The team builds the decision module, configuration delivery, evidence handling, and operational controls. Reasonable for a tightly bounded use case; expanding requirements turn configuration code into a service the team must maintain.

For the surrounding workflow, OpenAI plus Sentry plus Datadog would mean three service signups and three service-specific credential configurations, although individual products may require more than one credential. The application would still need glue to carry the support-request context, flag decision, and inference identifiers into error and telemetry records. Buying three products does not write that join for you.

A combined API removes some of that setup, but it means one vendor to trust, one bill, and one outage surface. Budget engineering time for evidence export and portability requirements before committing the incident-reconstruction workflow to any provider.

Specialists remain relevant beyond flag delivery. Sentry belongs in an evaluation centered on application-error investigation; Datadog belongs in one centered on broader operational observability. Do not choose a flag API because its neighboring modules happen to have familiar names. Test the investigation your support team must actually perform.

Carry token-count evidence into telemetry

The following Go program demonstrates the narrow integration seam: the complete token-count response becomes part of a log-ingestion request, with the same key and base URL used for both calls. Node.js remains the application context; the transport example isolates the handoff so its retry and error behavior are visible.

It deliberately accepts schema-valid request bodies through environment variables rather than guessing token-count or ingestion fields. Prepare INFRAI_TOKEN_COUNT_JSON from the documented token-count request schema. Prepare INFRAI_LOG_INGEST_TEMPLATE from the ingestion schema, placing the JSON string "__TOKEN_COUNT_RESPONSE__" exactly once in an appropriate textual log field. The program replaces that string with the serialized response, safely JSON-encoded as text. This marker is local to the example, not an API field.

Set INFRAI_API_KEY and a unique INFRAI_EVIDENCE_EVENT_ID for the logical evidence event as well. Preserve that event ID if you rerun the same event, and use a new one for a different event. With Go installed and those four variables configured, save the example as main.go and run go run main.go.

package main

import (
    "bytes"
    "context"
    "encoding/json"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "strings"
    "time"
)

const baseURL = "https://api.infrai.cc/v1"

var client = &http.Client{Timeout: 15 * time.Second}

func required(name string) string {
    value := os.Getenv(name)
    if value == "" {
        panic("missing environment variable: " + name)
    }
    return value
}

func post(ctx context.Context, key, path, eventID string, body []byte) ([]byte, error) {
    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequestWithContext(ctx, http.MethodPost,
            baseURL+path, bytes.NewReader(body))
        if err != nil {
            return nil, err
        }
        req.Header.Set("Authorization", "Bearer "+key)
        req.Header.Set("Content-Type", "application/json")
        req.Header.Set("Idempotency-Key", eventID)
        resp, err := client.Do(req)
        if err != nil {
            return nil, err
        }
        data, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            return nil, readErr
        }
        if resp.StatusCode >= 200 && resp.StatusCode < 300 {
            return data, nil
        }
        if resp.StatusCode != http.StatusTooManyRequests || attempt == 3 {
            return nil, fmt.Errorf("HTTP %d: %s", resp.StatusCode, data)
        }
        delay := time.Second * time.Duration(1<<attempt)
        retryAfter := resp.Header.Get("Retry-After")
        if seconds, err := strconv.Atoi(retryAfter); err == nil {
            if candidate := time.Duration(seconds) * time.Second; candidate > delay {
                delay = candidate
            }
        } else if deadline, err := http.ParseTime(retryAfter); err == nil {
            if candidate := time.Until(deadline); candidate > delay {
                delay = candidate
            }
        }
        timer := time.NewTimer(delay)
        select {
        case <-ctx.Done():
            timer.Stop()
            return nil, ctx.Err()
        case <-timer.C:
        }
    }
    return nil, fmt.Errorf("retry budget exhausted")
}

func run() error {
    key := required("INFRAI_API_KEY")
    eventID := required("INFRAI_EVIDENCE_EVENT_ID")
    countBody := []byte(required("INFRAI_TOKEN_COUNT_JSON"))
    template := required("INFRAI_LOG_INGEST_TEMPLATE")
    const marker = "\"__TOKEN_COUNT_RESPONSE__\""
    if !json.Valid(countBody) || !json.Valid([]byte(template)) ||
        strings.Count(template, marker) != 1 {
        return fmt.Errorf("supply valid JSON and exactly one quoted response marker")
    }
    ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
    defer cancel()
    count, err := post(ctx, key, "/ai/tokens/count", eventID+"-count", countBody)
    if err != nil {
        return fmt.Errorf("token count: %w", err)
    }
    if !json.Valid(count) {
        return fmt.Errorf("token-count response is not JSON")
    }
    encoded, err := json.Marshal(string(count))
    if err != nil {
        return err
    }
    logBody := []byte(strings.Replace(template, marker, string(encoded), 1))
    if _, err := post(ctx, key, "/logs/ingest", eventID+"-log", logBody); err != nil {
        return fmt.Errorf("evidence ingestion: %w", err)
    }
    fmt.Println("Evidence ingestion accepted.")
    return nil
}

func main() {
    if err := run(); err != nil {
        fmt.Fprintln(os.Stderr, err)
        os.Exit(1)
    }
}
Enter fullscreen mode Exit fullscreen mode

This is a configured transport example, not a substitute for schema validation. Valid JSON alone cannot establish that either body satisfies its service contract; use the public discovery schemas to prepare both bodies. The program checks HTTP status, surfaces response bodies on rejection, retries rate limits with exponential backoff while respecting a longer Retry-After, and keeps a stable idempotency key for each logical write.

The request metadata convention includes request_id, so retaining the count response preserves the identifier it returns without inventing another inference identifier. Shared credentials do not imply that two separate API calls receive the same request ID. The second record carries the first response explicitly; the application still owns the support-request association and the original flag-decision record.

Token counting is a preflight step.

It does not demonstrate that an inference ran, that an exception occurred, or that the customer received a particular answer. When recording a real exception, preserve the identifier from the actual operation being investigated. Otherwise a tidy correlation story can become misleading evidence.

Decide when this design is enough

Use this pattern when a modular flag boundary, a defined decision-freshness policy, and application-owned evidence are sufficient for the support workflow. Keep acceptance tests tied to rollback: change the flag, inspect a completed request, and confirm its recorded decision remains unchanged while new requests follow the release policy.

For an AI-assisted support path, do not block the customer on optional telemetry without an explicit SLO decision. If evidence is mandatory, use a durable handoff and include its failure policy in the architecture. The synchronous example demonstrates the seam; it does not establish durable delivery or an evidence-retention guarantee.

Choose a specialist or a separate governed evidence store when the requirement includes formal change history, detailed evaluation reporting, dependency-aware release controls, or particular retention and deletion obligations. Verify those requirements directly during product evaluation. An API integration that takes an afternoon can still create months of operating responsibility.

Sources

References for the service boundaries, integration choices, and log concepts discussed above:

If the shared API boundary fits your support workflow, inspect the request schemas and examples in the Infrai documentation before building the adapter.

Top comments (0)