Brazilian "Rinha de Backend" challenge #1 - How To Fail

Father's Day at Brazil I received a message from a friend of mine talking about a tech challenge. A quite simple one: Write a API, that has at least 2 replicas, a load balancer and a database, so someone can try to tear it down with a Stress Test.

I though to myself. That is the next level of CRUD developer, so let's try it out.

The initial idea, by which the commit history made it clear I mistakenly called it "finished", was:

• NodeJS Express
• Postgres
  • Indexes for FTS (gin(to_tsvector(lan, field));)
  • Stack being a Array field
• Batching
  • All read requests to database

I knew this configuration was far away from aiming for any TOP 20 but I had confidence it wouldn't fail that much.

After my first NGINX configuration and Docker-Compose YAML created I boldly applied to the challenge.

Sitting at the top of my confidence and not-so worried about it I noticed the competitors publishing screenshots of their own benchmarks.I had realized how much I had underestimated it.

Gatling installed locally, lets give it a shot. To be fair...How bad can my application perform, right?

Lets not talk about how disappointed I got but how worthy it was to try the tool locally.

So, what was happening?

Besides a bunch of premature closed connections, no healthy upstream, 60000ms timeouts, there was no way I my application was bottlenecking that much...

I tried a lot of crazy stuff:

• Clustering - via PM2 or native cluster module.
• Closing slow requests early
• Adjusted some business rules which were off
• Reconfigured NGINX
• Changing LIBUV process.env.UV_THREADPOOL_SIZE = os.cpus().length

I was very confident I could get at least into 20% of success...Come on...

I had to do something. I was totally blind and didn't knew where to start.

After running a watch "docker stats" I was able to finally get a grasp on it. My database was being hammered.

Time to take it seriously. More than competing with my fellow colleagues I wanted to compete with myself. I knew I could do better. It was a long time I got so excited about something I knew nothing about.

In that late Sunday I already knew Monday would be one of those hyperfocus ADHD days.

Next chapter