We launched 10k+ code changes to production. It took us 4.5h and it was (surprisingly) problem-free. This was our recipe.

Two days ago we launched 10k+ code changes to production. Although it was intense, it was problem-free, unlike some other deployments. The deployment process lasted 4.5 hours, with us following a 10-step process:

1. We organized everyone mission-critical on standby and were ready for the deployment to last much more than anticipated.
2. We did it overnight to disturb as few users as possible. For us, this meant fewer variables to worry about.
3. We created a Google spreadsheet with all the scenarios that had to work perfectly. For every scenario, we defined 3 phases: development, dev QA, product QA.
4. We wrote a deployment guide, simulated deployment locally multiple times, and iterated over it.
5. Before launch, we deployed a sandbox, restored production databases there, and verified that everything worked.
6. We then deployed a new production environment next to the active one. We had to set up many things and wanted to verify everything was done correctly.
7. Once verified, we created new backups on the old production and restored them on the new production.
8. We rerouted the traffic to the new production.
9. We kept the old production on standby if things went terribly wrong (you never know). Since everything worked, we turned it off after a day.
10. We all went to sleep with our ringtones on maximum in case of any errors.

If you find my content interesting, follow me on twitter. We can share half-baked ideas and discuss engineering challenges.