We deploy our order API to thirty servers with an Ansible playbook that runs with serial: 5. For each batch it takes the servers out of the HAProxy pool with a delegated task, installs the new package, restarts the service, waits for the health port with wait_for and a sixty second timeout, and puts the servers back in the pool. Five at a time, we told ourselves, limited a bad release to a sixth of the fleet.
On a Thursday in October a release added a product cache that is warmed when the service starts. On our servers the warm up took between fifty and ninety seconds. In the first batch one server answered inside sixty seconds and four did not, so wait_for failed on those four. Ansible marked them failed and ran nothing more on them, including the task that would have put them back in the pool. The fifth server went back in, and Ansible moved on to the second batch. When the play ended, twenty two of thirty servers were out of the pool, all running the new version perfectly well, and eight servers were carrying the lunchtime traffic of thirty.
That is how serial behaves. A batch is a unit of scheduling, not of judgement. Ansible ends the play when every host in a batch has failed, and as long as one host per batch gets through, it carries on to the next. To stop sooner you set max_fail_percentage, and the percentage has to be exceeded, not merely reached. We had never set it.
We put the servers back by hand within ten minutes. The playbook now sets max_fail_percentage: 0, so the first failed host ends the rollout and every later batch is left alone. The health timeout is now three times the slowest start the service logged last month. A rescue block records why a host failed, leaves it out of the pool on purpose and then fails it again, so the rollout still stops. The last task of the play compares the number of servers in the pool with the number in the inventory and fails if they differ.
We had sized our batches for a bad release. What we got was a bad health check, and Ansible applied it to every batch in turn because we had never said how much failure was enough.
– Sergey Shinder
Top comments (0)