DEV is a Ruby on Rails application deployed on Heroku servers.
The other day Molly Struve, our resident SRE sorceress, noticed we were having some memory troubles. They didn't look like memory leaks, since they plateaued over time. But they were obviously worth investigating.
After investigating a bit I remembered how in the past I happened to switch a Rails app from the standard memory allocator to jemalloc.
The jemalloc library is an alternative memory allocator that can be used by apps (Redis ships with it) which works better with memory fragmentation in a multithreading environment.
We researched that, activated it and this was the result:
The slight decrease you see before the red bar is due to deployments which for obvious reasons free some occupied memory.
How do I do the same in my Rails app(s)?
The prerequisite is that you're using multiple threads with Rails, otherwise it won't help much.
If you're using Heroku you can follow these instructions.
If not, you can take inspiration from this tutorial and recompile your Ruby and activate jemalloc
support.
Why does this happen?
Nate Berkopec has an extensive explanation in his post Malloc Can Double Multi-threaded Ruby Program Memory Usage but long story short: the Ruby virtual machine and the default memory allocator (malloc
) aren't great at talking to each other when multithreading is involved because of the way they are both designed.
Are there any downsides?
Potentially yes, as it would make your app less portable (it doesn't work on Windows) and there are options with a smaller footprint that are being explored by Ruby core, like using malloc_trim().
My suggestion is to test it :-)
Top comments (6)
Great job Rhymes and Molly!
I would also suggest few things to track down the issue further:
Creating a rails app with just one endpoint and all the gems used in DEV, to determine the normal from the abnormal memory usage.
Using a memory profiler and determining which endpoints account for the majority of the memory usage.
After pinpointing the bottlenecks, you might consider switching their implementations into a faster one like C/Go/Rust (I suspect the serialization and deserialization are the culprits).
All that above matters if there is some real financial pain, otherwise it's not worth pursuing 😁
Thanks a lot for the suggestions!
The gem
derailed_benchmarks
also helps:schneems / derailed_benchmarks
Go faster, off the Rails - Benchmarks for your whole Rails app
Derailed Benchmarks
A series of things you can use to benchmark a Rails or Ruby app.
Compatibility/Requirements
This gem has been tested and is known to work with Rails 3.2+ using Ruby 2.1+. Some commands may work on older versions of Ruby, but not all commands are supported.
For some benchmarks, not all, you'll need to verify you have a working version of curl on your OS:
Install
Put this in your gemfile:
Then run
$ bundle install
.While executing your commands you may need to use
bundle exec
before typing the command.To use all profiling methods available also add:
You must be using Ruby 2.1+ to install these libraries. If you're on an older version of Ruby, what are you waiting for?
Use
There are…
Looks like PHP internally uses
malloc
implementation?php.net/manual/en/internals2.memor...
I'm not 100% sure about it, as I'm unfamiliar with PHP. I went as far as this file github.com/php/php-src/blob/master... which contains the declaration of the list of allocators you linked.
same story happened to our nodejs project (mostly processing jpeg image buffers with libvips), jemalloc managed to save about half of memory usage.
(using github.com/gaffneyc/heroku-buildpa...)
I've been using Jemalloc here and it solved all my Sidekiq eating infinite memory issues. It is just amazing!