Sergey Shinder - DEV Community

Sergey Shinder

Sep 24

The incident ended and the spare never came back

#sergeyshinder #sre #reliability #incident

2 min read

Sergey Shinder

Sep 24

The deploy took four minutes and the release took six days

#sergeyshinder #releaseengineering #database #automation

2 min read

Sergey Shinder

Sep 24

A third of our calls came from an address our carrier had never been given

#sergeyshinder #api #integration #networking

2 min read

Sergey Shinder

Sep 24

We spent thirteen weeks about to buy a bigger database

#sergeyshinder #observability #tracing #performance

2 min read

Sergey Shinder

Sep 24

The node was ready in seventy seconds and our pod was not

#sergeyshinder #kubernetes #docker #autoscaling

2 min read

Sergey Shinder

Sep 24

We forked a module for one line and owned all of it

#sergeyshinder #terraform #iac #devops

2 min read

Sergey Shinder

Sep 24

Thirty seven tests failed the day our runner stopped running them in order

#sergeyshinder #cicd #testing #devops

2 min read

Sergey Shinder

Sep 23

The order was committed and nothing else ever heard about it

#sergeyshinder #sre #reliability #messaging

2 min read

Sergey Shinder

Sep 23

The fix we backported to our oldest branch did nothing

#sergeyshinder #releaseengineering #versioning #automation

2 min read

Sergey Shinder

Sep 23

Twenty workers refreshed one token and invalidated each other

#sergeyshinder #api #oauth #integration

1

2 min read

Sergey Shinder

Sep 23

The requests our customers gave up on were in none of our numbers

#sergeyshinder #observability #monitoring #latency

2 min read

Sergey Shinder

Sep 23

Two in three of our internal calls left the zone they started in

#sergeyshinder #kubernetes #networking #cloud

2 min read

Sergey Shinder

Sep 23

Somebody's laptop upgraded our state file

#sergeyshinder #terraform #iac #devops

2 min read

Sergey Shinder

Sep 23

Our integration tests were talking to our partner's live system

#sergeyshinder #cicd #testing #integration

2 min read

Sergey Shinder

Sep 22

Our deploys were also our restart policy

#sergeyshinder #sre #reliability #kubernetes

2 min read

Sergey Shinder

Sep 22

We shipped the version that could not receive the next one

#sergeyshinder #releaseengineering #automation #deployment

2 min read

Sergey Shinder

Sep 22

We were the unreliable half of the integration

#sergeyshinder #api #integration #backend

2 min read

Sergey Shinder

Sep 22

Every question we asked our logs took six minutes to answer

#sergeyshinder #observability #logging #elasticsearch

2 min read

Sergey Shinder

Sep 22

Our cluster was full at nineteen percent CPU

#sergeyshinder #kubernetes #containers #capacity

2 min read

Sergey Shinder

Sep 22

One workspace planned clean and eleven were never asked

#sergeyshinder #terraform #iac #testing

2 min read

Sergey Shinder

Sep 21

The cheapest check in our pipeline ran last

#sergeyshinder #cicd #kubernetes #devops

2 min read

Sergey Shinder

Sep 20

Our rate limit punished everyone except the customer causing the problem

#sergeyshinder #sre #reliability #ratelimiting

2 min read

Sergey Shinder

Sep 20

Nine teams were taking turns on one staging environment

#sergeyshinder #releaseengineering #devops #automation

1

2 min read

Sergey Shinder

Sep 20

We never told the partner which version of their API we wanted

#sergeyshinder #api #integration #versioning

2 min read

Sergey Shinder

Sep 20

We spent the first twenty three minutes of an outage reading staging's graphs

#sergeyshinder #observability #grafana #monitoring

2 min read

Sergey Shinder

Sep 20

Our cluster had kept every job it ever finished

#sergeyshinder #kubernetes #containers #devops

1

2 min read

Sergey Shinder

Sep 20

An empty list in our module meant every bucket

#sergeyshinder #terraform #iac #aws

2 min read

Sergey Shinder

Sep 20

Every merge in the company depended on the dev cluster staying up

#sergeyshinder #cicd #testing #devops

1

2 min read

Sergey Shinder

Sep 19

Nobody had timed the part that comes after the restore

#sergeyshinder #sre #reliability #backups

2 min read

Sergey Shinder

Sep 19

Five percent of our traffic was not five percent of our callers

#sergeyshinder #releaseengineering #deployment #devops

2 min read

Sergey Shinder

Sep 19

Our totals were wrong by pennies and the pennies were in the parser

#sergeyshinder #api #integration #json

2 min read

Sergey Shinder

Sep 19

One new label multiplied our metrics by every order we take

#sergeyshinder #observability #prometheus #monitoring

2 min read

Sergey Shinder

Sep 19

The configuration we changed in June arrived one pod at a time in August

#sergeyshinder #kubernetes #containers #configuration

2 min read

Sergey Shinder

Sep 19

Terraform adopted the database and set it back to our defaults

#sergeyshinder #terraform #iac #devops

1

2 min read

Sergey Shinder

Sep 19

Our pipeline tested the files we touched and none of the code we broke

#sergeyshinder #cicd #monorepo #devops

2 min read

Sergey Shinder

Sep 18

The queue had no limit so our timeouts meant nothing

#sergeyshinder #sre #reliability #architecture

2 min read

Sergey Shinder

Sep 18

The migration finished in a second and the rollout ran for eleven minutes

#sergeyshinder #releaseengineering #database #devops

2 min read

Sergey Shinder

Sep 18

Their certificate worked in every browser and failed in our service

#sergeyshinder #api #integration #tls

2 min read

Sergey Shinder

Sep 18

Averaging our percentiles gave us a number nobody had experienced

#sergeyshinder #observability #monitoring #metrics

2 min read

Sergey Shinder

Sep 18

Our pods slept for sixty milliseconds out of every hundred

#sergeyshinder #kubernetes #containers #performance

2 min read

Sergey Shinder

Sep 18

Deleting one subnet from a list rebuilt six of them

#sergeyshinder #terraform #iac #devops

2 min read

Sergey Shinder

Sep 18

The same commit built us two different images

#sergeyshinder #cicd #docker #devops

2 min read

Sergey Shinder

Sep 17

We announced read only mode and could not even sign in

#sergeyshinder #sre #reliability #database

2 min read

Sergey Shinder

Sep 17

A one line fix took three weeks of unreleased work to production

#sergeyshinder #releaseengineering #git #devops

2 min read

Sergey Shinder

Sep 17

On the second of January we started overwriting last year's orders

#sergeyshinder #api #integration #database

1

2 min read

Sergey Shinder

Sep 17

The timeline we built from the logs was physically impossible

#sergeyshinder #observability #logging #sre

2 min read

Sergey Shinder

Sep 17

Half of our new capacity could not run our image

#sergeyshinder #kubernetes #containers #devops

2 min read

Sergey Shinder

Sep 17

Every apply we ran was a small one on purpose

#sergeyshinder #terraform #iac #devops

2 min read

Sergey Shinder

Sep 17

Two formatters disagreed and the branch built itself all night

#sergeyshinder #cicd #githubactions #automation

2 min read

Sergey Shinder

Sep 16

The admission webhook rejected every change, including the one that would fix it

#sergeyshinder #kubernetes #sre #reliability

2 min read

Sergey Shinder

Sep 16

Eight months in production and no automation had ever touched it

#sergeyshinder #releaseengineering #automation #devops

2 min read

Sergey Shinder

Sep 16

Our partner calls failed more often the quieter it got

#sergeyshinder #api #integration #http

2 min read

Sergey Shinder

Sep 16

Eighty five percent full gave us seventeen minutes

#sergeyshinder #monitoring #alerting #sre

2 min read

Sergey Shinder

Sep 16

The debug copy of our service was taking one request in three

#sergeyshinder #kubernetes #containers #devops

2 min read

Sergey Shinder

Sep 16

A firewall rule with two owners changed itself back every three hours

#sergeyshinder #terraform #iac #devops

2 min read

Sergey Shinder

Sep 16

Our test runner had one core and the race needed two

#sergeyshinder #cicd #testing #concurrency

2 min read

Sergey Shinder

Sep 15

Every new replica took twenty more database connections

#sergeyshinder #sre #postgres #reliability

2 min read

Sergey Shinder

Sep 15

A third of our releases went out through the emergency path

#sergeyshinder #releaseengineering #devops #cicd

1

2 min read

Sergey Shinder

Sep 15

Their sandbox accepted everything their production refused

#sergeyshinder #api #integration #testing

2 min read

Sergey Shinder

Sep 15

Nobody could tell me when the latency had started

#sergeyshinder #observability #monitoring #performance

2 min read

Writing Debut