Thank you Webpacker, Goodbye Webpacker

Original Post (Japanese) : ありがとう Webpacker さようなら Webpacker - クラウドワークス エンジニアブログ

Hello. I'm Okuto Oyama, an engineer at CrowdWorks. I'm on a team that leads in eliminating technical debt.

This article is replace Webpacker with Simpacker and webpack.

As an assumption, our products have been running on monolithic Ruby on Rails applications for over a decade. Currently, we are trying to break the tight coupling between front-end and back-end as part of our technical debt resolution.

I hope this is helpful for anyone who wants to separate front-end responsibilities from Webpacker!

What is Webpacker?

Webpacker is the official library for webpack front-end development in Ruby on Rails applications. The advantage of this library is that it provides a generic configuration for those who are not familiar with the webpack build system, making it easy to quickly develop modern front-ends.

For more information on CrowdWorks' implementation of Webpacker, please see our previous blog posts.

CrowdWorks流！Webpacker活用術 - クラウドワークス エンジニアブログ

Why quitting Webpacker?

I have decided to quit Webpacker, and I will give the reasons why I have decided to quit.

Because Webpacker was declared retired in January of this year.

• There is no need to continue using Webpacker as the next RC version will not be officially released in the future.
  • There is an option to use Shakapacker
  • However, since we plan to move away from Rails-dependent front-end development at CrowdWorks, this is not the right choice for us.

Because the modules that depend on Webpacker are hidden.

• The internal Webpacker dependencies are hidden, making it hard to see what is being used.
  • Babel settings were not included in CrowdWorks package.json, causing some build failures.
  • Debugging itself is not easy.
    • I tried the v14 series of stylelint postcss must be included, but even if the lint passed, the webpack build failed.
      • I wanted to find the cause, but at first glance I couldn't figure out what was wrong with the configuration.
  • By migrating to bare webpack, we can think about the configuration ourselves.

Because it is hard to do performance tuning

• Not only webpack, but also Rails assets:precompile builds are done by CI.
  • We are extending resource classes because CI and deployments are failing due to the long time it takes.
  • Not enough fine-tuning of webpack.
  • It may be possible to reduce this problem by updating Webpacker itself.
    • The webpack included in the stable version of Webpacker used to be v4.

In light of the above, we decided to eliminate the use of Webpacker itself and allow the front-end responsibilities to be handled by the front-end.

Incidentally, CrowdWorks would like to migrate to a front-end framework in the future, and if everything can run on that framework, the removal of Webpacker itself will be unnecessary. However, since the transition to a framework is not yet in sight, we have decided that using webpack is the right way to maintain the status quo.

Organizing what to do

First of all, when I quit Webpacker, I organized what I was going to do. There have been several articles about quitting Webpacker in the past, so I used them as a reference.

I mainly referred to pixiv's article and iCARE's article. I would like to take this opportunity to thank them.

• webpack configuration
  • Migrate Webpacker DSL (config/webpack) to webpack.config.js
  • Install necessary plugins and modules.
  • Configure webpack-dev-server
• Remove Webpacker
  • Remove from gem
  • Make config/webpacker.yml obsolete
• Implementing webpack view helpers in Rails
  • Create helper code
  • Create test code
• Docker, CI configuration changes
  • Incorporate webpack build mechanism into CI
  • Adding webpack processing to docker-compose configuration
  • Staging CI checks
  • Checking docker-compose behavior
• Checking the behavior of docker-compose in local staging
  • Local screen checks (webpack build-related screens only)
  • Screen check at staging (only screens related to webpack builds)

Examine the Webpacker implementation

The first thing I did was investigate the Webpacker implementation.

First, I tried to output the Webpacker configuration to get a rough idea of how the whole thing works.

The way to do this is quite simple. I put the logs in console.dir(webpackConfig, { depth: null }); at the end of module.exports in each environment's DSL file placed in config/webpack. I have tried this. I can't go into the details of the function-based code, but this gave me a rough idea of the general shape of the code.

The following is the webpack configuration included in Webpacker.

The next step is to investigate the detailed Webpacker implementation based on the results.

https://github.com/rails/webpacker/tree/5-x-stable

The version of Webpacker used at CrowdWorks was version 5, so we checked the 5-x-stable branch. The part I was interested in was webpack.config.js, so I just looked at its source.

• webpacker/package/environments/ ... Environment settings. It contains basic webpack settings.
• webpacker/package/rules/ ... List of rules for loaders. Only refer to the ones you use.

You should also check the required package.json, babel.config.js, postcss.config.js.

Compare build results between Webpacker and webpack alone

I experimented to see if Webpacker can be built by itself while referring to the internal implementation.

In CrowdWorks, the front-end tools (such as Storybook) are in a separate hierarchy of directories, so I installed a new one there and set up webpack to reference the front-end code in the application.

After confirming that the build went through, I compared the differences between the results built in the staging environment and the results built with webpack alone.

Since CrowdWorks uses CircleCI for our application build pipeline, we saved the built files in artifacts.

- run:
  name: Compress Artifacts
  command: tar -cvzf assets-precompile.tar public/packs
- store_artifacts:
  name: Uploading artifacts - store results of assets:precompile
  path: assets-precompile.tar

We reviewed the missing configuration, the number of files, and the differences in the contents of the files, and finally, we were able to get the files to almost match ¹.

Incorporate webpack build results into your Rails application

After confirming that the build results are OK, the next step is to incorporate them into the Rails application.

Webpacker uses webpack-assets-manifest to create manifest.json, which is then delivered to View helper in Rails.

I needed to create an alternative View helper to replace Webpacker, but decided to use the Simpacker gem rather than implement the helper on my own.

Unlike Webpacker, Simpacker does not manage any configuration on the webpack side and is only involved with the path to the output manifest.json. The responsibilities are nicely divided, and the configuration itself is very simple.

I have been using Split Chunks since I was using Webpacker, so I decided to replace it by introducing the following helper.

def javascript_packs_with_chunks_tag(*names, **options)
  paths = names.flat_map{ |name| simpacker_context.manifest.lookup!("entrypoints", name, "js") }.uniq
  javascript_include_tag(*paths, **options)
end

def stylesheet_packs_with_chunks_tag(*names, **options)
  paths = names.flat_map{ |name| simpacker_context.manifest.lookup!("entrypoints", name, "css") }.uniq
  stylesheet_link_tag(*paths, **options)
rescue Simpacker::Manifest::MissingEntryError
  # If you don't extract css, you get an error, but don't return anything
end

Make webpack-dev-server work

Webpacker also includes webpack-dev-server, so we made it work as well. Since I had Simpacker installed, I only had to change the devServer part in webpack.config.js to make it work with npm scripts.

The problem with getting webpack-dev-server to work was that the included version was a 3 series that didn't work well with the same settings as Webpacker.

Rather than investigating the cause in detail, I thought that if it was for development only and didn't affect production, I should just update it to the latest version, so I updated to the latest 4 series and migrated the part about devServer settings.

It works with this as a result. If you're looking to move on from the Webpacker 5, here's what you need to know.

Check the behavior on CI

As mentioned above, deployment of CrowdWorks is managed by CircleCI. Because it's a monolithic application, the front-end build mechanism is built into it, and we'll review the settings when we switch to webpack to make sure it doesn't fail.

Webpacker used to build webpack at assets:precompile time, but since it no longer does so, we need to add our webpack build commands.

While reviewing the configuration, I also included a workflow uploading source map files to Rollbar for the front-end error monitoring system. However, for some reason, I caught an event that prevented the uploading of the files.

After investigating the cause, I found that it was due to the --ignore-optional option that I had added when doing the yarn install.

This was because the source map files generated by webpack depended on optional dependencies, and thus were not generated properly without the required modules. Therefore, we decided to remove this option and let it install.

Ask engineers to check if it works in their local environment

Since CrowdWorks uses Docker in its development, we asked our engineers to check if it works with Simpacker + webpack.

Docker startup and build results were fine, but we found that Hot Module Reloads in webpack-dev-server were not working properly. After that, we fixed it while checking the operation with my team.

Conduct a mob review session

Since the changes made to the PR were very significant this time, we conducted a mob review session to reduce the prerequisite knowledge and cognitive load among my team.

Since I had been working on this project for a long time, I was able to look back on what I had done, and the causes of some parts that were unclear as to their operation were clarified. Some of the omissions found in this review have also been corrected to reflect.

Formulate Architecture Decision Records

We have been keeping Architecture Decision Records (ADR) since September of this year to keep the reasons for architecture selection.

This initiative was not in place when we discontinued Webpacker, but since we were going to adopt Simpacker in this transition, we decided to develop an ADR for it and have it reviewed by our engineers.

Incidentally, this is the first document after the ADR initiative was introduced.

Release

It's been a long process, but now that we have all the necessary responses, we're finally releasing it. Even in the staging environment, I had verified it many times and thought there would be no problem with the release, but I had shared it with all the engineers beforehand just in case.

I was very nervous about the release because it was a very big change, but after the release was completed, we determined that there were no problems with operation and no alerts and that users would not be affected, so we reported the completion and shared the operating procedures again.

I was in a daze for a while, maybe glad that I released it successfully...

Reflection

We look back on our efforts to quit Webpacker this time.

Good

• I was able to complete the migration to Simpacker + webpack itself.
• We were able to split webpack management and build responsibilities from Rails.
• We were able to resolve the EOL situation.

Problem

• adding their own npm packages that were included with Webpacker
  • Try ⇒ We will update and prune it
• As the npm packages were being updated at the same time, there were many conflicts in package.json, making it difficult to rebase
  • We should have verified the module with the module itself added and released it first
  • Try ⇒ Make a mechanism to verify build diffs (to build psychological safety for future updates)
• Migration work was being handled by @okuto_oyama alone
  • The work was being done by one person, @okutoyama, who had been doing the research singlehandedly, and was also fixing the problem
    • @bugfire also helped me with some of the stuck areas
  • The work could have been divided among the team since it was related to the development of the entire engineering team
  • Try ⇒ After investigating, you don't start as is, but set tasks one by one after you get a general perspective

Conclusion

We started our research in April of this year, and although there was a period during which we had to stop due to other work within the team, we were finally able to migrate and release it on November 24. We are very happy that we were able to finish it before the end of the year.

I've had a few struggles with Webpacker, but it's also the fact that it's been a mainstay of CrowdWorks' modern front-end development for a long time. Thank you for fighting with us so far.

Webpacker脱却記事も多いので、フロントエンド人材が育ったら引き剥がすことがそこまで難しくなさそうなのもポイントです。

Webpackerをフロントエンド環境の補助輪と考えて、将来存在が足かせになったタイミングでWebpackerのレイヤは剥がしてもらえばよいと考えています。

メンテ不能になったフロントエンド環境を立て直す話 - クラウドワークス エンジニアブログ

After rereading articles about rebuilding the front-end environment that had existed in the past, I personally felt a lingering sense of joy that we had reached a point where it was OK to eliminate this layer.

Although I was able to quit Webpacker, I hope to continue working on improving the build flow without compromising the developer experience, including upgrading webpack.

References

• 今日から簡単！Webpacker 完全脱出ガイド - pixiv inside
• RailsからWebpackerを外してpureなwebpack構成にしてみる その3 | Mission-Street.
• フロントエンド原理主義者が目論んだ脱webpacker. webpacker+vueのプロジェクトをwebpack+vueにする | by Tomoya Komiyama | スタディスト開発ブログ
• Steb by Step で剥がす Webpacker - 30歳からのプログラミング
• WebpackerをやめるならWebpackManifestというgemが便利、という話 | Raksul ENGINEERING
• Rails: Webpacker→jsbundling-rails+webpackアップグレード手順（翻訳）｜TechRacho by BPS株式会社
• モノリシックな Ruby on Rails サービスからの Webpacker の剥がし方 | 働くひとと組織の健康を創る iCARE

1. Where there were differences in some of the build results, we decided to allow them because we were able to confirm and determine that there were no problems with the display and operation. ↩