DEV Community

rhymes for The DEV Team

Posted on

How we decreased our memory usage with jemalloc

DEV is a Ruby on Rails application deployed on Heroku servers.

The other day Molly Struve, our resident SRE sorceress, noticed we were having some memory troubles. They didn't look like memory leaks, since they plateaued over time. But they were obviously worth investigating.

After investigating a bit I remembered how in the past I happened to switch a Rails app from the standard memory allocator to jemalloc.

The jemalloc library is an alternative memory allocator that can be used by apps (Redis ships with it) which works better with memory fragmentation in a multithreading environment.

We researched that, activated it and this was the result:

Alt Text

The slight decrease you see before the red bar is due to deployments which for obvious reasons free some occupied memory.

How do I do the same in my Rails app(s)?

The prerequisite is that you're using multiple threads with Rails, otherwise it won't help much.

If you're using Heroku you can follow these instructions.

If not, you can take inspiration from this tutorial and recompile your Ruby and activate jemalloc support.

Why does this happen?

Nate Berkopec has an extensive explanation in his post Malloc Can Double Multi-threaded Ruby Program Memory Usage but long story short: the Ruby virtual machine and the default memory allocator (malloc) aren't great at talking to each other when multithreading is involved because of the way they are both designed.

Are there any downsides?

Potentially yes, as it would make your app less portable (it doesn't work on Windows) and there are options with a smaller footprint that are being explored by Ruby core, like using malloc_trim().

My suggestion is to test it :-)

Top comments (6)

Collapse
 
yaser profile image
Yaser Al-Najjar • Edited

Great job Rhymes and Molly!

I would also suggest few things to track down the issue further:

  1. Creating a rails app with just one endpoint and all the gems used in DEV, to determine the normal from the abnormal memory usage.

  2. Using a memory profiler and determining which endpoints account for the majority of the memory usage.

  3. After pinpointing the bottlenecks, you might consider switching their implementations into a faster one like C/Go/Rust (I suspect the serialization and deserialization are the culprits).

  4. All that above matters if there is some real financial pain, otherwise it's not worth pursuing 😁

Collapse
 
rhymes profile image
rhymes

Thanks a lot for the suggestions!

The gem derailed_benchmarks also helps:

GitHub logo schneems / derailed_benchmarks

Go faster, off the Rails - Benchmarks for your whole Rails app

Derailed Benchmarks

A series of things you can use to benchmark a Rails or Ruby app.

Build Status Help Contribute to Open Source

Compatibility/Requirements

This gem has been tested and is known to work with Rails 3.2+ using Ruby 2.1+. Some commands may work on older versions of Ruby, but not all commands are supported.

For some benchmarks, not all, you'll need to verify you have a working version of curl on your OS:

$ which curl
/usr/bin/curl
$ curl -V
curl 7.37.1 #

Install

Put this in your gemfile:

gem 'derailed_benchmarks', group: :development

Then run $ bundle install.

While executing your commands you may need to use bundle exec before typing the command.

To use all profiling methods available also add:

gem 'stackprof', group: :development

You must be using Ruby 2.1+ to install these libraries. If you're on an older version of Ruby, what are you waiting for?

Use

There are…




Collapse
 
zubairmohsin33 profile image
Zubair Mohsin

the memory management API looks very much like libc's malloc implementation

Looks like PHP internally uses malloc implementation?

php.net/manual/en/internals2.memor...

Collapse
 
rhymes profile image
rhymes

I'm not 100% sure about it, as I'm unfamiliar with PHP. I went as far as this file github.com/php/php-src/blob/master... which contains the declaration of the list of allocators you linked.

Collapse
 
lk9100 profile image
Chinese broccoli • Edited

same story happened to our nodejs project (mostly processing jpeg image buffers with libvips), jemalloc managed to save about half of memory usage.

(using github.com/gaffneyc/heroku-buildpa...)

Collapse
 
rodolfobandeira profile image
Rodolfo

I've been using Jemalloc here and it solved all my Sidekiq eating infinite memory issues. It is just amazing!