Sergey Shinder - DEV Community

Sergey Shinder

Nov 4

One curly apostrophe made every delivery text three messages long

#sergeyshinder #integration #sms #api

2 min read

Sergey Shinder

Nov 4

Our warehouse link was full and the graph said it was using three hundred megabits

#sergeyshinder #monitoring #prometheus #networking

2 min read

Sergey Shinder

Nov 4

Our workers wedged after a broker failover and Kubernetes never ran the health check in their image

#sergeyshinder #kubernetes #docker #devops

2 min read

Sergey Shinder

Nov 4

We started shipping to Norway and Ansible wrote it into our config as false

#sergeyshinder #ansible #yaml #iac

2 min read

Sergey Shinder

Nov 4

Three services shared one deploy queue and the middle one never deployed

#sergeyshinder #cicd #githubactions #deployment

2 min read

Sergey Shinder

Nov 3

Our address lookups took one second or three and the service said it had answered in forty milliseconds

#sergeyshinder #sre #linux #performance

2 min read

Sergey Shinder

Nov 3

Our deploy script restarted twelve servers in three minutes and systemctl said yes to every one

#sergeyshinder #systemd #linux #deployment

2 min read

Sergey Shinder

Nov 3

Our supplier moved its price file to a CDN and every accented product name came back garbled

#sergeyshinder #python #integration #api

2 min read

Sergey Shinder

Nov 3

We upgraded Prometheus and our latency alert stopped matching the bucket it was written for

#sergeyshinder #prometheus #monitoring #observability

2 min read

Sergey Shinder

Nov 3

Our new carrier key ended with a newline and every pod refused to send it

#sergeyshinder #kubernetes #go #devops

2 min read

Sergey Shinder

Nov 3

Ansible ran our run once task four times and gave our servers four signing keys

#sergeyshinder #ansible #automation #iac

2 min read

Sergey Shinder

Nov 3

Our deploy job got an empty image tag because the runner thought it was a secret

#sergeyshinder #cicd #githubactions #helm

1

2 min read

Sergey Shinder

Nov 2

Our label service ran out of local ports at four hundred and seventy requests a second

#sergeyshinder #sre #linux #networking

1

2 min read

Sergey Shinder

Nov 2

Helm reported a successful release while its data job was still failing in the first region

#sergeyshinder #helm #kubernetes #releaseengineering

1

2 min read

Sergey Shinder

Nov 2

One apostrophe in a delivery address ended our despatch advice in the middle of a segment

#sergeyshinder #integration #edi #api

2 min read

Sergey Shinder

Nov 2

Our disk alert waited for ninety seven percent and Postgres stopped at ninety five

#sergeyshinder #monitoring #linux #postgres

2

1

2 min read

Sergey Shinder

Nov 2

Our third search replica waited six hours for a node in a zone our node group did not cover

#sergeyshinder #kubernetes #storage #aws

2 min read

Sergey Shinder

Nov 2

We changed the user data on our integration server and Terraform restarted it without running a line

#sergeyshinder #terraform #aws #iac

2 min read

Sergey Shinder

Nov 2

Our deploy workflow waited for CI to finish and never asked whether it had passed

#sergeyshinder #cicd #githubactions #devops

2 min read

Sergey Shinder

Nov 1

Our Redis replica spent three days restarting the same sync

#sergeyshinder #redis #sre #reliability

1

2 min read

Sergey Shinder

Nov 1

We moved one Go variable and every copy of our new CLI was told to upgrade

#sergeyshinder #go #releaseengineering #automation

1

2 min read

Sergey Shinder

Nov 1

The schema registry approved a change that only worked if we deployed first

#sergeyshinder #kafka #integration #avro

2 min read

Sergey Shinder

Nov 1

One slow count query took every metric our dispatch service had off the dashboard

#sergeyshinder #prometheus #observability #monitoring

2 min read

Sergey Shinder

Nov 1

Our statement job ran out of memory at seventy percent and Kubernetes started it again from the first customer

#sergeyshinder #kubernetes #batch #containers

2 min read

Sergey Shinder

Nov 1

We passed null to say use the default and Terraform switched off deletion protection

#sergeyshinder #terraform #aws #iac

2 min read

Sergey Shinder

Nov 1

One failing job in our nightly matrix cancelled the eleven that would have told us more

#sergeyshinder #cicd #githubactions #testing

2 min read

Sergey Shinder

Oct 30

Our notification service ran out of memory at two in the morning and stayed half alive until nine

#sergeyshinder #sre #java #kubernetes

1

2 min read

Sergey Shinder

Oct 30

Every deploy broke our portal for anyone who already had it open

#sergeyshinder #releaseengineering #frontend #aws

1

2 min read

Sergey Shinder

Oct 30

Our invoice links were valid for seven days and stopped working after an hour

#sergeyshinder #aws #integration #s3

2 min read

Sergey Shinder

Oct 30

Every info line our Python services wrote arrived in Cloud Logging as an error

#sergeyshinder #observability #python #gcp

2 min read

Sergey Shinder

Oct 30

We added a sleep before shutdown and our pods ran out of time to shut down

#sergeyshinder #kubernetes #containers #kafka

2 min read

Sergey Shinder

Oct 30

Ansible changed our nginx config on forty servers and reloaded it on twenty nine

#sergeyshinder #ansible #nginx #iac

1

2 min read

Sergey Shinder

Oct 30

Our migrations passed in CI as a superuser and failed in the first new database we built

#sergeyshinder #cicd #postgres #testing

2 min read

Sergey Shinder

Oct 30

CloudWatch had a slow hour and our delivery slot service stopped because it could not finish writing a log line

#sergeyshinder #sre #docker #logging

1

2 min read

Sergey Shinder

Oct 30

Helm upgraded our operator and kept its old CRD, and the cluster threw the new setting away

#sergeyshinder #releaseengineering #kubernetes #helm

2 min read

Sergey Shinder

Oct 30

Our calls to the carrier failed for twenty minutes after every deploy and then fixed themselves

#sergeyshinder #grpc #integration #networking

2 min read

Sergey Shinder

Oct 28

We put the error rate in our alert's labels and it never stayed the same alert long enough to fire

#sergeyshinder #observability #prometheus #alerting

2 min read

Sergey Shinder

Oct 28

Our document store took forty one minutes to start because Kubernetes rechecked every file on its disk

#sergeyshinder #kubernetes #storage #containers

2 min read

Sergey Shinder

Oct 28

One secret in our task definition hid the database change sitting next to it

#sergeyshinder #terraform #aws #iac

2 min read

Sergey Shinder

Oct 28

A hundred and eighty six tests passed on every laptop and CI never collected one of them

#sergeyshinder #cicd #python #testing

2 min read

Sergey Shinder

Oct 28

One queue nobody was reading filled the disk and RabbitMQ stopped every publisher we had

#sergeyshinder #sre #rabbitmq #messaging

2 min read

Sergey Shinder

Oct 28

Our rollout finished in two minutes and the new pods started crashing in the fifth

#sergeyshinder #releaseengineering #kubernetes #node

2 min read

Sergey Shinder

Oct 27

We streamed our claims file to save memory and the insurer's gateway asked how long it was

#sergeyshinder #api #integration #python

1

2 min read

Sergey Shinder

Oct 27

We stretched one Alertmanager timer to quiet a disk warning and our payment alert waited behind it

#sergeyshinder #observability #prometheus #alerting

1

2 min read

Sergey Shinder

Oct 27

Our ingress saw every visitor as one of our own nodes and let them reach the admin pages

#sergeyshinder #kubernetes #networking #security

1

2 min read

Sergey Shinder

Oct 27

Ansible turned off password logins on every server and sshd read another file first

#sergeyshinder #ansible #linux #security

1

2 min read

Sergey Shinder

Oct 27

Our nightly image rebuild stopped because nobody had pushed to the repository for sixty days

#sergeyshinder #cicd #githubactions #docker

1

2 min read

Sergey Shinder

Oct 19

A cleanup job that listed every cart key paused Redis for the whole site

#sergeyshinder #sre #redis #performance

1

2 min read

Sergey Shinder

Oct 19

Our signing certificate expired and took every installer we had ever shipped with it

#sergeyshinder #releaseengineering #windows #security

1

2 min read

Sergey Shinder

Oct 19

Our new API gateway lowercased every header and no partner could place an order

#sergeyshinder #api #integration #aws

1

2 min read

Sergey Shinder

Oct 19

Our backup dashboard stayed green for five weeks after the job had moved

#sergeyshinder #observability #prometheus #backups

1

2 min read

Sergey Shinder

Oct 19

Our log shipper moved to the host network and could no longer find the cluster

#sergeyshinder #kubernetes #dns #logging

2

2 min read

Sergey Shinder

Oct 19

We recreated our CI role with the same name and lost control of our encryption keys

#sergeyshinder #terraform #aws #security

2

2 min read

Sergey Shinder

Oct 19

Our deploy form had an unticked box and every deploy rebuilt the search index

#sergeyshinder #cicd #githubactions #automation

2

2 min read

Sergey Shinder

Oct 19

One export job switched off the query timeout for strangers

#sergeyshinder #sre #postgres #database

1

2 min read

Sergey Shinder

Oct 16

Our routing table outgrew kubectl apply and half a release went out

#sergeyshinder #releaseengineering #kubernetes #kubectl

2 min read

Sergey Shinder

Oct 16

Every supplier file with a space in its name was refused as access denied

#sergeyshinder #api #integration #aws

2 min read

Sergey Shinder

Oct 16

Kubernetes knew why checkout failed at three in the morning and forgot by four

#sergeyshinder #observability #kubernetes #monitoring

1

2 min read

Sergey Shinder

Oct 16

Our autoscaler stopped scaling the day we added a sidecar without a CPU request

#sergeyshinder #kubernetes #autoscaling #containers

2 min read

Sergey Shinder

Oct 16

Terraform said no changes and our Lambda kept running last month's code

#sergeyshinder #terraform #aws #serverless

2 min read

Writing Debut