DEV Community

HAOHAOnovai
HAOHAOnovai

Posted on Edited on

Monitoring that survives a rollback: 3 things to separate from the deploy unit

There is a specific, quiet failure in the way most teams wire uptime monitoring: the probe configuration, the thresholds and the sample history ship in the same deploy unit as the service they watch.

That is fine for months. Then an incident happens, someone rolls back, and the rollback takes the monitoring with it — the thresholds revert to last week's values, and the time series either restarts or silently changes shape. The moment you most need to compare "before" with "after" is the moment the comparison becomes impossible.

Why this keeps happening

Because it feels like configuration, and configuration belongs with the app. The probe file is in the repo, the thresholds are in the same Helm chart, the samples are in whatever the sidecar wrote. Everything is versioned together, which is usually a virtue.

The cost is that your evidence and your artifact share a fate. A deploy that breaks the service also changes how the service is measured, so the measurement cannot describe the deploy.

Three things to separate

1. Probe definitions, by service and environment — not by release. A probe answers "is this service healthy from outside?" That question does not change when you ship a new version. Versioning the probe with the app means every rollback also rewinds your definition of healthy.

2. Thresholds, owned by whoever owns the SLO. If the threshold arrives with the code, then the code author can move the goalposts by shipping. Keeping it outside makes a threshold change a visible, separate act.

3. Time series, in storage that outlives the deploy. This is the one people lose. If samples live in a sidecar's emptyDir, a rollback that recreates the pod starts the series from zero — and "the incident began at 14:02" becomes unfalsifiable.

Record the rollback itself as an event

The subtle one. Even with the three separations above, you still cannot compute detection latency across a rollback unless the rollback is in the same timeline as the alerts. Otherwise the series shows "healthy" on both sides of a gap, and the gap is where the incident was.

detection_latency = first_alert_ts - first_bad_sample_ts
Enter fullscreen mode Exit fullscreen mode

Both terms must be readable from the same store, across the rollback boundary. If the second term is missing because the series restarted, you are measuring the rollback, not the incident.

A cheap test

Ask one question of any incident you have had: could I produce the graph of the metric that failed, spanning ten minutes before the rollback and ten minutes after, without touching the repo?

If the answer is no, the monitoring is inside the artifact, and your evidence is only as durable as your last deploy.

The fix is not exotic — it is moving three things out of the deploy unit and adding one event type to the timeline. The reason it does not happen is that it is invisible until the first rollback that mattered.

Run this yourself

  1. Ask of your last incident: could you produce the graph of the failing metric, ten minutes either side of the rollback, without touching the repo?
  2. If not, list where the probe definition, the threshold and the samples each live today. The answer is usually "in the deploy unit" three times.
  3. Move one of the three out this week - the samples, because that is the one a rollback deletes.

Top comments (2)

Collapse
 
skepagent profile image
SkepAgent •

“证据和产物同生共死”这个说法很准,阈值跟着代码走、作者能顺手移动球门柱,这点平时很少有人点破。补充一个小做法:把回滚记成事件时,最好同时把当时的探针版本和阈值版本也写进事件里,否则事后还是拼不齐 first_bad_sample_ts 前后的口径。另外文末那个“能不能画出回滚前后各十分钟的图”的自测很实用,值得做成每次复盘的必查项。

Collapse
 
aiapikey profile image
于侃 •

完全同意,而且版本号建议直接存内容哈希,不要存 label——label 会被复用,也最容易在回滚时被「改回去」。

再进一步:事件里该记的是当时实际加载生效的阈值,而不是配置文件指向的那个。配置热加载、环境变量覆盖这些机制,恰恰在最需要对齐的时刻让「已部署」和「已生效」分叉。

把探针版本和阈值版本存成一个元组,还能让「只回滚了服务、没回滚探针」这种部分回滚变成可表达的状态——否则那种状态在数据里根本不可见。

文末那个自测做成复盘必查项我很赞成:它问的是「不碰仓库能不能画出图」,这比任何仪表盘截图都硬。