Earlier this year, deploys of my team's main application began failing with this error:
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
This is a Rails app with an AngularJS front-end currently being converted to React. In the months leading up to those failures, deploy times had steadily increased. Before they began failing, our longest deploys took 24+ minutes. π± Here's how we fixed the issue and what I learned about the cause.
Attempted Fixes
The --max_old_space_size
Setting
We increased Node.js's memory limit to 2GB by setting --max_old_space_size=2048
as recommended in several Stack Overflow posts and Github issues. While this worked for many others, it did not solve our problem. Deploys continued to fail.
Node.js Upgrade
We next upgraded the app's Node.js version from 8 to 12 to take advantage of this feature:
This update will configure the JavaScript heap size based on available memory instead of using defaults that were set by V8 for use with browsers. In previous releases, unless configured, V8 defaulted to limiting the max heap size to 700 MB or 1400MB on 32 and 64-bit platforms respectively. Configuring the heap size based on available memory ensures that Node.js does not try to use more memory than is available and terminating when its memory is exhausted. (source)
Upgrading Node.js unblocked our deploys for several weeks. However, during that time, we continued converting our AngularJS code to React and added new features in React. Deploys took longer and longer, and after a while, they began failing again.
The Fix
Given the attempted fixes above and with the help of infrastructure monitoring already in place, we were pretty sure that we weren't running out of memory on our deploy server. As it turns out, the root cause for this issue was in our Webpacker configuration.
Our webpacker.yml
contained this:
default: &default
source_path: app-web
source_entry_path: react
...
Because of the way our app is structured, this meant we were telling Webpacker to process ALL of our React and Redux-related files, which were increasing in number with every sprint. As I researched the deploy failures, I learned about a helpful of rule of thumb about Webpacker from Ross Kaffenberger's blog:
If any file in Webpacker's "packs" directory does not also have a corresponding
javascript_pack_tag
in your application, then you're overpacking.
Based on this rule, I should've seen just one file in our packs
directory. What I saw, though, was essentially a replica of the entire structure of our /app-web/react
directory. We were overpacking.
Ultimately, we moved only the two necessary files into a startup
directory and reconfigured webpacker.yml
to use that as its entry point:
default: &default
source_path: app-web
source_entry_path: react/startup
...
What I Learned
What is Webpacker, and what does it do?
Webpacker is a gem which allows Rails apps to use webpack to process and bundle assets, particularly JavaScript.
According to its documentation, webpack "is a static module bundler for modern JavaScript applications. When webpack processes your application, it internally builds a dependency graph which maps every module your project needs and generates one or more bundles."
Okay, cool. But what does that actually mean?
Webpack basically does the work of figuring out what depends on what in your application to generate the minimum "bundles" of assets required to run your app. You include these minimum packs in your application - in Rails, like below - so the app can load with the necessary assets already compiled.
<%= javascript_pack_tag 'application' %>
See this article for a much more in-depth intro to what webpack actually does and why module bundlers are needed.
Why was our configuration wrong?
Since webpack builds a dependency graph based on a specified entry point, the greater the number of items in that entry point, the more processing time and resources needed. Because our configuration told Webpacker to process ALL of our React files, this required more time and server resources as we added more files to the React directory.
So, basically, the idea was to not ask Webpacker to process every single file in our React application, but only the entry points to the React app (aka the files that have corresponding javascript_pack_tag
s), so that they and their immediate dependencies would be ready on initial application load.
Impact
This fix unblocked our deploys and dramatically reduced our deploy times and resource usage on our deploy server.
Deploy Time | Max Deploy CPU Usage | Max Deploy Memory Usage | |
---|---|---|---|
Before Fix | > 24 min | ~90% | ~2.2GB |
After Fix | 10 min | ~60% | ~0.28GB |
So, lesson learned - don't overpack with Webpacker! π§³
Photo by Erwan Hesry on Unsplash
Top comments (2)
Filed. This is the sort of problem that worries me. And now I have a candidate solution for that rainy day. Cheers!
Glad you found this potentially useful! Hopefully not something youβll run into! π