Sergey Shinder - DEV Community

Sergey Shinder

Oct 15

Our standby received every change for five hours and applied none of them

#sergeyshinder #sre #postgres #database

2 min read

Sergey Shinder

Oct 15

A deploy killed halfway left Helm refusing every release after it

#sergeyshinder #releaseengineering #helm #kubernetes

2 min read

Sergey Shinder

Oct 15

Our sync sent null for every field it had not loaded and the partner deleted them

#sergeyshinder #api #integration #json

2 min read

Sergey Shinder

Oct 15

Every log line over sixteen kilobytes arrived in pieces and none of them counted as an error

#sergeyshinder #observability #logging #kubernetes

2 min read

Sergey Shinder

Oct 15

Our delivery photos stopped uploading the week we moved to ingress-nginx

#sergeyshinder #kubernetes #nginx #containers

2 min read

Sergey Shinder

Oct 15

One depends_on line taught us to stop reading our Terraform plans

#sergeyshinder #terraform #iac #aws

2 min read

Sergey Shinder

Oct 15

Our integration tests skipped themselves on every Dependabot pull request

#sergeyshinder #cicd #githubactions #testing

2 min read

Sergey Shinder

Oct 14

Our quote service stopped at lunchtime with its CPU at two percent

#sergeyshinder #sre #java #concurrency

1

2 min read

Sergey Shinder

Oct 14

We rolled back the image and kept the configuration that broke it

#sergeyshinder #releaseengineering #kubernetes #rollback

1

2 min read

Sergey Shinder

Oct 14

Our price job put back the stock our stock job had just sold out

#sergeyshinder #api #integration #http

1

2 min read

Sergey Shinder

Oct 14

Our dashboard counted every failed refund except the first one on each pod

#sergeyshinder #observability #prometheus #metrics

1

2 min read

Sergey Shinder

Oct 14

cert-manager renewed our certificate and the pods kept serving the old one

#sergeyshinder #kubernetes #tls #containers

2

1

2 min read

Sergey Shinder

Oct 14

Attaching one policy to our CI role took it away from every node in the cluster

#sergeyshinder #terraform #aws #iam

1

2 min read

Sergey Shinder

Oct 14

Our release workflow tagged three versions that were never published

#sergeyshinder #cicd #githubactions #automation

1

2 min read

Sergey Shinder

Oct 13

Our lock expired while the first worker was still writing invoices

#sergeyshinder #sre #redis #distributedsystems

4

2 min read

Sergey Shinder

Oct 13

I renamed an Argo CD application and it deleted everything it managed

#sergeyshinder #releaseengineering #argocd #gitops

2

2 min read

Sergey Shinder

Oct 13

One webhook in ten failed its signature check because of a trailing zero

#sergeyshinder #api #webhooks #security

4

1

2 min read

Sergey Shinder

Oct 13

Maintenance on our staging cluster muted the alert for our public website

#sergeyshinder #observability #prometheus #alerting

2

2 min read

Sergey Shinder

Oct 13

A service called redis gave every new pod a port it could not parse

#sergeyshinder #kubernetes #containers #configuration

1

2 min read

Sergey Shinder

Oct 13

Our ninety day retention rule had not deleted a recording in nineteen months

#sergeyshinder #terraform #aws #s3

1

2 min read

Sergey Shinder

Oct 13

A hidden folder never reached our artifact and our app links opened in the browser

#sergeyshinder #cicd #githubactions #mobile

1

2 min read

Sergey Shinder

Oct 12

One transaction left open for three weeks stopped every write in our database

#sergeyshinder #sre #postgres #reliability

2

1

2 min read

Sergey Shinder

Oct 12

Our deploy script kept every value except the ones the new chart added

#sergeyshinder #releaseengineering #helm #kubernetes

1

2 min read

Sergey Shinder

Oct 12

We replayed failed payouts after the provider had forgotten them

#sergeyshinder #api #payments #integration

1

2 min read

Sergey Shinder

Oct 12

Every deploy told our autoscaler that traffic had gone up forty times

#sergeyshinder #observability #prometheus #kubernetes

1

2 min read

Sergey Shinder

Oct 12

Our month end reports failed because the container had sixty four megabytes of shared memory

#sergeyshinder #kubernetes #postgres #containers

1

2 min read

Sergey Shinder

Oct 12

Amazon published a new image and our next apply rebuilt the SFTP server

#sergeyshinder #terraform #iac #aws

1

2 min read

Sergey Shinder

Oct 12

A customer found our registry token in the history of our image

#sergeyshinder #cicd #docker #security

3

1

2 min read

Sergey Shinder

Oct 2

Our users were logged out for two weeks before Redis refused to write anything

#sergeyshinder #sre #redis #reliability

2 min read

Sergey Shinder

Oct 2

Our beta became the version every new customer installed

#sergeyshinder #releaseengineering #npm #javascript

2 min read

Sergey Shinder

Oct 2

The SMS provider told us when to come back and we heard now

#sergeyshinder #api #integration #http

2 min read

Sergey Shinder

Oct 2

Our error graph went blank every time we zoomed in on the outage

#sergeyshinder #observability #prometheus #grafana

1

2 min read

Sergey Shinder

Oct 2

A reporting job outranked our ingress controllers and the scheduler agreed

#sergeyshinder #kubernetes #scheduling #containers

2 min read

Sergey Shinder

Oct 2

Our bucket policy let the office in and locked out every server we own

#sergeyshinder #terraform #aws #iam

2 min read

Sergey Shinder

Oct 2

Our invoices lost their accents when we moved to a new CI system

#sergeyshinder #cicd #git #docker

1

2 min read

Sergey Shinder

Oct 1

A connector we switched off in May kept every change our database made

#sergeyshinder #sre #postgres #reliability

2 min read

Sergey Shinder

Oct 1

A migration merged on Tuesday was never going to run in production

#sergeyshinder #releaseengineering #database #migrations

2 min read

Sergey Shinder

Oct 1

Our HR sync decided José had left and then hired him again

#sergeyshinder #api #integration #unicode

2 min read

Sergey Shinder

Oct 1

Our synthetic check spent five weeks testing the login page

#sergeyshinder #observability #monitoring #synthetics

2 min read

Sergey Shinder

Oct 1

Our node pool rotation hit Docker Hub's limit and new nodes could not start their pods

#sergeyshinder #kubernetes #docker #containers

1

2 min read

Sergey Shinder

Oct 1

A refactor that touched no database changed our database password

#sergeyshinder #terraform #iac #postgres

2 min read

Sergey Shinder

Oct 1

We made checkout shallow and every build became version 0.1.0

#sergeyshinder #cicd #git #versioning

2 min read

Sergey Shinder

Sep 30

We added a fourth server and made our cluster easier to break

#sergeyshinder #sre #reliability #distributedsystems

2 min read

Sergey Shinder

Sep 30

Our new region launched with thirty one flags we had switched off everywhere else

#sergeyshinder #releaseengineering #featureflags #automation

2 min read

Sergey Shinder

Sep 30

Our customs broker changed an XML prefix and every shipment stayed on the dock

#sergeyshinder #api #xml #integration

2 min read

Sergey Shinder

Sep 30

Thirty four pods each failed a little and our alert judged them one at a time

#sergeyshinder #observability #monitoring #alerting

2 min read

Sergey Shinder

Sep 30

The node was fifty eight percent full and could not create a single file

#sergeyshinder #kubernetes #docker #linux

2 min read

Sergey Shinder

Sep 30

Every apply during the sale shrank our web tier back to six instances

#sergeyshinder #terraform #iac #aws

2 min read

Sergey Shinder

Sep 30

Pip found a newer version of our internal package on the public index

#sergeyshinder #cicd #python #security

2 min read

Sergey Shinder

Sep 29

Every cache entry we warmed at deploy expired in the same minute

#sergeyshinder #sre #reliability #caching

2

1

2 min read

Sergey Shinder

Sep 29

Our deploy tool decided that 2.9.0 was newer than 2.10.0

#sergeyshinder #releaseengineering #versioning #automation

1

2 min read

Sergey Shinder

Sep 29

Every order paid in yen looked a hundred times underpaid

#sergeyshinder #api #payments #integration

1

2 min read

Sergey Shinder

Sep 29

Which service lost its logs depended on who wrote first after midnight

#sergeyshinder #observability #logging #elasticsearch

1

2 min read

Sergey Shinder

Sep 29

Every pod on the node was ready and none of them could open a connection

#sergeyshinder #kubernetes #linux #networking

1

2 min read

Sergey Shinder

Sep 29

Our disaster recovery copies were in the same region as the originals

#sergeyshinder #terraform #iac #aws

2 min read

Sergey Shinder

Sep 29

A hung test held the only runner allowed to deploy our hotfix

#sergeyshinder #cicd #githubactions #testing

1

2 min read

Sergey Shinder

Sep 28

The partner recovered in four minutes and our circuit breaker kept them out for three hours

#sergeyshinder #sre #reliability #resilience

2 min read

Sergey Shinder

Sep 28

We switched traffic to green and blue kept doing the work

#sergeyshinder #releaseengineering #deployment #devops

2 min read

Sergey Shinder

Sep 28

We moved our webhook and the partner's POSTs arrived as empty GETs

#sergeyshinder #api #webhooks #integration

1

2 min read

Sergey Shinder

Sep 28

Four in ten of our requests were our own health checks

#sergeyshinder #observability #monitoring #sre

2

2 min read

Writing Debut