DEV Community

Sergey Shinder profile picture

Sergey Shinder

I am Sergey Shinder

Joined Joined on 
One PDF endpoint in our async service made every other endpoint on the worker wait

One PDF endpoint in our async service made every other endpoint on the worker wait

1
Comments
2 min read
Our hotfix took fifty six minutes to deploy because the depot screens never hung up

Our hotfix took fifty six minutes to deploy because the depot screens never hung up

1
Comments
2 min read
Two customers bought the same box of screws within five minutes and SQS delivered one sale

Two customers bought the same box of screws within five minutes and SQS delivered one sale

1
Comments
2 min read
Our label service logged every carrier rejection for three weeks and the reason was always [Object]

Our label service logged every carrier rejection for three weeks and the reason was always [Object]

6
Comments 1
2 min read
Our import worker was killed for memory and the last log line we had was thirty thousand rows old

Our import worker was killed for memory and the last log line we had was thirty thousand rows old

1
Comments
2 min read
We moved our roles to include_role and the emergency certificate run skipped every task

We moved our roles to include_role and the emergency certificate run skipped every task

1
Comments
2 min read
Our tests imported the repository and passed while the package we deployed was missing eleven templates

Our tests imported the repository and passed while the package we deployed was missing eleven templates

1
Comments
2 min read
Every deploy gave a full share of traffic to JVMs that had not compiled anything yet

Every deploy gave a full share of traffic to JVMs that had not compiled anything yet

1
Comments
2 min read
We set maxUnavailable to zero, the platform team set a pod quota, and no release could start

We set maxUnavailable to zero, the platform team set a pod quota, and no release could start

1
Comments
2 min read
Our thirty second timeout let one call to the customs portal run for forty minutes

Our thirty second timeout let one call to the customs portal run for forty minutes

1
Comments
2 min read
Our logs put one order's email failure under another order's number because a pooled thread remembered it

Our logs put one order's email failure under another order's number because a pooled thread remembered it

1
Comments
2 min read
We moved our cut-off service to a scratch image and it started working in UTC

We moved our cut-off service to a scratch image and it started working in UTC

1
Comments
2 min read
Ansible added our deploy user to one group and took it out of every other

Ansible added our deploy user to one group and took it out of every other

1
Comments
2 min read
Our order service built against a library snapshot someone had deployed from a feature branch

Our order service built against a library snapshot someone had deployed from a feature branch

1
Comments
2 min read
We looked up our new hostname four minutes too early and our resolver remembered that it did not exist

We looked up our new hostname four minutes too early and our resolver remembered that it did not exist

1
Comments
2 min read
We added one nullable column and every pod failed its order lookups until each connection had seen the error

We added one nullable column and every pod failed its order lookups until each connection had seen the error

1
Comments 1
2 min read
Our carrier switched to American date order and we only noticed on the thirteenth

Our carrier switched to American date order and we only noticed on the thirteenth

1
Comments
2 min read
Our new paging route for payments sat below a broader one and Alertmanager never reached it

Our new paging route for payments sat below a broader one and Alertmanager never reached it

1
Comments
2 min read
We autoscaled our Java pricing service on memory and it grew to twenty pods and stayed there

We autoscaled our Java pricing service on memory and it grew to twenty pods and stayed there

2
Comments
2 min read
Ansible changed our feed service's unit file and the restart kept the old one

Ansible changed our feed service's unit file and the restart kept the old one

1
Comments
2 min read
Our Node 20 image crashed on start because Docker copied the build agent's node_modules over its own

Our Node 20 image crashed on start because Docker copied the build agent's node_modules over its own

1
Comments
2 min read
Our reports hung for fifteen minutes every morning on connections the firewall had already forgotten

Our reports hung for fifteen minutes every morning on connections the firewall had already forgotten

Comments
2 min read
We deleted our old reminder CronJob from Git and Argo CD kept running it for six weeks

We deleted our old reminder CronJob from Git and Argo CD kept running it for six weeks

Comments 1
2 min read
One customer's price list grew past four megabytes and our gRPC client refused to read it

One customer's price list grew past four megabytes and our gRPC client refused to read it

Comments
2 min read
Our order graph went flat at peak because the kernel was dropping StatsD packets

Our order graph went flat at peak because the kernel was dropping StatsD packets

Comments
2 min read
Our snapshot init container asked for six gigabytes and every pod kept the reservation

Our snapshot init container asked for six gigabytes and every pod kept the reservation

Comments
2 min read
Ansible put our export job in /etc/cron.d and Debian's cron never read the file

Ansible put our export job in /etc/cron.d and Debian's cron never read the file

Comments
2 min read
Our images carried an empty commit label for five months because the ARG sat above FROM

Our images carried an empty commit label for five months because the ARG sat above FROM

Comments
2 min read
One finance report ran our database host out of memory and Postgres dropped every connection

One finance report ran our database host out of memory and Postgres dropped every connection

Comments
2 min read
We removed replicas from our manifest for the autoscaler and checkout dropped to one pod

We removed replicas from our manifest for the autoscaler and checkout dropped to one pod

Comments 1
2 min read
RabbitMQ took our booking job back after thirty minutes and gave it to the other consumer

RabbitMQ took our booking job back after thirty minutes and gave it to the other consumer

Comments
2 min read
Our latency alert summed away the bucket label and never returned a value

Our latency alert summed away the bucket label and never returned a value

5
Comments 1
2 min read
Our new Compose project took a carrier's address range and the host stopped reaching them

Our new Compose project took a carrier's address range and the host stopped reaching them

Comments
2 min read
Ansible updated our servers five at a time and kept going after most of each batch failed

Ansible updated our servers five at a time and kept going after most of each batch failed

Comments
2 min read
We moved our portal build to Vite and stopped type checking for four months

We moved our portal build to Vite and stopped type checking for four months

Comments
2 min read
We gave our notification executor fifty threads and it sent everything on one

We gave our notification executor fifty threads and it sent everything on one

1
Comments
2 min read
We moved our Redis chart to Argo CD and every new render gave it a new password

We moved our Redis chart to Argo CD and every new render gave it a new password

1
Comments 1
2 min read
Our carrier rotated our API secret and the longer one broke our header in two

Our carrier rotated our API secret and the longer one broke our header in two

1
Comments
2 min read
Our request graph showed thousands of calls a second because gunicorn had four counters

Our request graph showed thousands of calls a second because gunicorn had four counters

1
Comments
2 min read
We scaled our pricing service to twelve pods and the first four kept every request

We scaled our pricing service to twelve pods and the first four kept every request

1
Comments
2 min read
We moved our reporting servers to Leeds and Ansible gave them the Leeds memory setting

We moved our reporting servers to Leeds and Ansible gave them the Leeds memory setting

2
Comments
2 min read
Our two person rule for production deploys let people approve their own runs

Our two person rule for production deploys let people approve their own runs

1
Comments
2 min read
One slow report kept marking our web servers down for ten seconds at a time

One slow report kept marking our web servers down for ten seconds at a time

1
Comments
2 min read
We fixed our label client in 0.0.15 and npm update kept every service on 0.0.14

We fixed our label client in 0.0.15 and npm update kept every service on 0.0.14

1
Comments
2 min read
Our largest partner's access token outgrew a header buffer and every call came back 400

Our largest partner's access token outgrew a header buffer and every call came back 400

1
Comments
2 min read
Our uptime check stayed green for forty minutes because CloudFront was answering it

Our uptime check stayed green for forty minutes because CloudFront was answering it

1
Comments 1
2 min read
We renamed a container port and our Service quietly dropped every new pod

We renamed a container port and our Service quietly dropped every new pod

1
Comments
2 min read
Ansible wrote yesterday's server addresses into our load balancer from its fact cache

Ansible wrote yesterday's server addresses into our load balancer from its fact cache

1
Comments
2 min read
Our nightly patch build reused one cached apt-get upgrade for seven months

Our nightly patch build reused one cached apt-get upgrade for seven months

1
Comments
2 min read
We gave our servers IPv6 addresses and every call to two of our carriers waited three seconds first

We gave our servers IPv6 addresses and every call to two of our carriers waited three seconds first

1
Comments
2 min read
Our index build timed out, the retry found it by name, and the release went out without it

Our index build timed out, the retry found it by name, and the release went out without it

1
Comments
2 min read
Our despatch job stopped at the first empty answer from SQS and left parcels for the next day

Our despatch job stopped at the first empty answer from SQS and left parcels for the next day

Comments
2 min read
Our restart alert resolved itself after ten minutes and the pod kept crashing until Monday

Our restart alert resolved itself after ten minutes and the pod kept crashing until Monday

Comments
2 min read
We downloaded our config in a postStart hook and one pod in four started without it

We downloaded our config in a postStart hook and one pod in four started without it

Comments
2 min read
We lowered our login rate limit during an attack and Terraform kept the old one

We lowered our login rate limit during an attack and Terraform kept the old one

Comments 1
2 min read
An advisory published that afternoon blocked our hotfix and every rollback we could build

An advisory published that afternoon blocked our hotfix and every rollback we could build

1
Comments 1
2 min read
We replaced our deploy keys with OIDC and every repository in the organisation could deploy our website

We replaced our deploy keys with OIDC and every repository in the organisation could deploy our website

1
Comments
2 min read
Our refund consumer was stopped for nine days and came back at the end of the topic

Our refund consumer was stopped for nine days and came back at the end of the topic

Comments 1
2 min read
We switched to squash merges and semantic-release found nothing to release for five weeks

We switched to squash merges and semantic-release found nothing to release for five weeks

Comments
2 min read
RabbitMQ confirmed every DHL Express label request and put none of them in a queue

RabbitMQ confirmed every DHL Express label request and put none of them in a queue

Comments
2 min read
loading...
⌂ 换网址