With AI workloads, SRE and platform teams are seeing their habits shaken up: a service that calls a large language model can respond in 200 milliseconds or in 20 seconds, cost one cent or one euro, and give two different answers to the same question. The reliability tools stay the same, but what you need to measure, and how you guarantee it, change significantly.
Most organisations already have mature SRE practices and internal platforms. The challenge is therefore no longer getting started, but absorbing a new category of services without multiplying dashboards or losing control of costs.
This article lists what actually changes, proposes a catalogue of suitable indicators and shows how a platform team can make these requirements available by default.
How do AI workloads fail differently?
A classic API fails outright: a 500 error, a timeout, an overflowing queue. An AI service adds subtler failure modes:
- Non-determinism. The same input does not always produce the same output, which complicates regression testing.
- Degraded quality without errors. A wrong but well-phrased answer returns a 200 code.
- Variable cost. The number of tokens depends on the question, the retrieved context and an agent’s loops. A badly designed prompt can double the bill without latency moving.
- External dependencies. Model providers, quotas, rate limits: all points of failure outside your scope.
- Drift. A change of model, prompt or data alters behaviour without any application code deployment.
- Scarce resources. GPUs are expensive and often shared; saturation shows up as queues rather than errors.
What field data shows
The State of SRE and Platform Engineering 2026 study published by Dynatrace gives some useful benchmarks. SLOs are very widespread (89% of respondents use them in at least some of their teams or systems), but around half feel that the growing number of data sources makes them harder to define. On the platform side, 74% already offer self-service observability dashboards, whereas only 40% build observability into all their deployments.
In other words, the foundations are there, but coverage remains partial. And an AI workload running in a poorly covered area quickly becomes a black box: you no longer know whether it behaves as it did during testing.
Which AI workloads SRE indicators should you track?
The four classic signals (latency, traffic, errors, saturation) remain the foundation. They are complemented by AI-specific indicators:
| Indicator | What it reveals | Example threshold |
|---|---|---|
| Time to first token | Perceived responsiveness when streaming | p95 below 1.5 s |
| Tokens per request | Drift in context or agent loops | Alert if +40% over 24 h |
| Cost per successful task | Real economic efficiency | Monthly budget per use case |
| Evaluation score | Answer quality on a sample | Above 0.85 |
| Provider error rate | Quotas, rate limits, outages | Below 1% |
| GPU queue | Inference saturation | Fewer than 10 requests waiting |
These thresholds are examples to adapt. What matters is choosing a few indicators tied to user experience, rather than adding dozens of metrics. For latency SLOs, a distribution-based approach is particularly well suited to responses of highly variable duration: see histogram metrics with OpenPipeline.
The platform team’s role: a paved road for AI
SREs and platform engineers increasingly work together: the study indicates that 73% of respondents share responsibilities between the two teams. For AI, this sharing takes a concrete form: the platform provides project templates in which instrumentation, SLOs, security guardrails and budgets are present from the moment the service is created. Developers do not have to reinvent monitoring for their agent; they inherit a common foundation.
This foundation must also cover the move from evaluation to production. Tests carried out during development and monitoring in operations often live in separate tools, which prevents you from checking that the system behaves in production as it did in testing. That is the common thread of our guide to end-to-end AI observability.
In practice: bringing an AI workload into your SRE approach
Step 1 — Classify AI workloads by criticality
Distinguish internal uses (coding assistant, ticket summaries) from uses exposed to customers or built into a business process. Only the latter justify strict SLOs and an on-call rota.
Step 2 — Write the SLO in the repository
Describe the objectives as versioned configuration. The format below is generic, to be transposed to your tool:
service: assistant-souscription
slo:
- nom: latence_premier_token
objectif: 99.0 # % of requests under the threshold
seuil_ms: 1500
fenetre: 28j
- nom: qualite_reponses
objectif: 95.0 # % of evaluated samples that pass
fenetre: 7j
budget:
cout_mensuel_max_eur: 4000
alerte_a: 0.8
Step 3 — Instrument by default in the deployment template
In the Kubernetes template provided by the platform, the standard OpenTelemetry variables ensure that every new service sends its traces to the right place:
env:
- name: OTEL_SERVICE_NAME
value: assistant-souscription
- name: OTEL_EXPORTER_OTLP_ENDPOINT
value: http://otel-collector.observabilite:4318
- name: OTEL_RESOURCE_ATTRIBUTES
value: team=souscription,criticite=haute
The collector address is an example. For inference nodes, also monitor the resources actually reserved by pods, a topic covered in effective CPU requests in Kubernetes.
Step 4 — Add an AI quality gate to the pipeline
Before every production release of a new prompt or model, run the evaluation set and block delivery if the score drops. Because results are probabilistic, compare an average over several runs rather than a single result, and complement this with monitoring in production.
Step 5 — Link costs and reliability
Display the error budget and the financial budget side by side. An agent that meets its latency SLO but triples its token consumption is an incident, even if it wakes nobody up.
Best practices
- Treat reliability, security and observability as a single requirement from the design stage.
- Limit AI SLOs to a few indicators close to the user.
- Version observability configuration and guardrails (« as code ») for auditing.
- Keep human approval for any high-impact automated action.
- Follow the OpenTelemetry GenAI semantic conventions to avoid home-grown formats.
Frequently asked questions
Does an AI service need specific SLOs?
Yes, for critical uses: on top of latency and errors, a quality objective measured through evaluation and a cost budget. Internal uses can make do with monitoring without on-call.
Who is responsible for answer quality, the SRE or the AI team?
The AI team defines the evaluators and improves the system; the SRE ensures they run continuously and that deviations trigger an alert. Responsibility is shared, hence the value of a common platform.
How do you avoid an explosion in the number of metrics?
Start from the four classic signals, add two or three AI indicators tied to user experience, and remove any dashboard that does not support a decision.
Conclusion
AI workloads do not make SRE practices obsolete; they force them to measure quality and cost on the same footing as availability. Teams that build these requirements into the platform, rather than leaving them to each project, gain consistency and peace of mind. The logical next step is to see how AI changes day-to-day reliability. The standard attributes are described in the OpenTelemetry GenAI semantic conventions.
Further reading: SRE best practices and platform engineering trends: How AI workloads raise demands on observability to meet reliability requirements (Dynatrace blog).
span.observer is an independent community blog, not affiliated with Dynatrace LLC or the projects mentioned. Trademarks belong to their respective owners. Tools change quickly: check commands and options in their official documentation.
Originally published on span.observer. French version: https://span.observer/fr/charges-ia-exigences-sre/.
Top comments (0)