The problem
Have you ever faced the issue of waiting what seems to be eternity while you are downloading dependencies after hitting that...
For further actions, you may consider blocking this person and/or reporting abuse
Although it is true other languages are compiled and JS is not, that's not the major reason behind the heaviness of node_modules.
Python, for example, is not a compiled programming language, but it comes with a mature standard library that solves most of your daily programming needs, so you need fewer third-party dependencies. That said, if you choose to install a Python library, you're getting the library's source code, not a binary.
Unlike JS and Python, Golang is a compiled language, but when you install a dependency, you also get the source code of that dependency
If you run
ls $GO_PATH/pkg/mod/github.com, you can see the source file of all go dependencies you're using in your projects, Not the binaries but the source code.I think the main reason for the heaviness of
node_modulescomes in two folds.Javascript's lack of a standard library has forced the community to depend solely on third-party solutions. Nodejs has a standard library, but it's not as mature as what you get from, say, a language like Python.
I think the Js community has generally adopted a culture of preferring someone else's code to theirs (I'm sometimes guilty). For example, [npmjs.com/package/is-string] package has 20+m weekly downloads. A package less than 30 lines of code (counting white spaces), and I'm almost certain that most people downloading this package could have just done
return typeof value === 'string'and then we haveis-string-and-not-blankwith 90+k weekly download. If you ask me I'd say that's ridiculousYou are correct, and if you read the blog, I did mention it that the reason why Golang projects are small even after downloading the source code, its because it has a rich standard, due to which most go dependencies will have less external dependencies, and your own project will also have less external dependencies.
Another thing you might have missed is that in languages like go and java, the dependency manager has very nice ability to not re-download a package which is already there. This is one of the faults of npm as a monorepo with multiple submodules might have the same dependency installed separately in each of the submodules, instead of doing it only once. In a language like Go, the dependency is managed centrally, which saves the storage of dependency.
Also, I slightly disagree on the part of python. I have definitely seen many python projects which uses multiple dependencies, and python dependencies are usually equal in size with node dependencies
@faizbshah I must have scrolled past the paragraph where you talked about go dependencies.
Great post
And I'm excited about what new runtimes like bun and deno are bringing to the game.
Same here!! Im very excited about the future of Bun. Its probably the most significant thing which happened in the ecosystem in the past 3-4 years
totally. i bet there's tons of redundant dependencies in even the simplest tree
Exactly. The redundancy is a big part why people have started to dislike npm as a package manager in recent times
If you want to avoid
node_modules, useesbuildor some other similar tool to only grab what you need.Additionally, abandon TypeScript, as type definition modules and the TypeScript facade itself are pretty chonky. TypeScript may add value for the developer, but it adds zero value to the end-user of the program.
There are a number of compilers for JavaScript out there, and they all have their trade-offs.
This is one of those problems where it feels like developer complain about the unintended consequences of their own convenience, but it is possible to write great JavaScript apps without massive bundles and dependency payloads.
Reduced bugs is a benefit to the user. Developer productivity and speed of delivery for features are benefits to the user. TS has multiple benefits for the end user. Things that make developers more efficient and more accurate are pretty much defacto good for end users unless the tools have tradeoffs that negatively impact users to a greater degree. Arguments about the impact on bundle size and time to interactive can certainly be had, but the assertion that TS has no benefit to users is just false.
Naw, nothing you said requires typescript to solve it. There are a number of approaches and languages which produce the same outcomes you’re describing, and when you combine that with the reality that typescript is a tedious layer on top of JavaScript, and that the majority of its ecosystem is still JavaScript, I stand my my correct assertion.
I do wish that the typescript zealotry would stop, it’s not constructive.
I think you are wrong here. Typescript is the best thing which was created for JS world from last 10 years. TypeScript will allow developers to improve their efficiency, improve confidence in the codebase and solution on which they are working. The codebase will become “understandable”, even through time, because of rich type system which JS misses. You are right in only one moment here - everything what was done in TS can be done in JS. But in large codebase pure JS become a pain in the butt and in most cases will take a lot more time and mental health to maintain. It’s always a point into TS money-box, benefits are outweigh a lot more, than possible drowback in performance (in correct setup and product architecture, with some development conventions the performance footprint can be can be very small)
I'm really excited for the enthusiasm you're bringing to the table, but please be careful not to confuse your enthusiasm and preferences for objective fact. I don't think JavaScript is lacking a type system because I've been programming long enough to know that a type system—especially one that isn't present at runtime—isn't necessary.
Some people prefer a type system, and some don't. I have my own preferences, and as long as we all remember that our preferences are only that—preferences—we can all be friends.
pnpm give a link instead copy same node_module library, so if you use same dependency multiple then it will be save your disk space
Keep your package.json simple as you can. In my experience with vite + pnpm is the best for speed a storage too.
Also the npm module creator responsibility is keep the module dependencies keep at minimal level.
For example, my really minimalist npm module: react-state-factory still using peer-dependency for avoide React version collision:
Even If someone just use
useStateFactoryhook, then do not need to install redux-saga and use-saga-reducer.So I think node_modules is also heavy if someone is check how many different React version is included ( after few React module used)
Yes, I mentioned pnpm in my blog post as well as in one of my comments. pnpm is far amazing than what npm has to offer!!
The next question then would be: Why, if any real project requires a JS compiler of some sort, why not use a real compiler instead? If we have to use a compiler either way, why not output a binary?
For any real project you need to use babel/TS/swc and a module system, then you can just use a real compiler instead.
And that's where WebAssembly comes in. The size of the node_modules folder is just one of the many problems that wasm will solve. Stop using JavaScript
To answer your question, I think the reason is simple: No one in the ecosystem is too interested in taking up that job. It took years before somebody actually become frustrated enough to create a runtime env better than Node (talking about Bun). A reason why the ecosystem don't want to deal with binaries is because that will introduce the dreaded problem of platform-level machine code dependency. And, unless there's a dedicated team behind that goal, no one wants to deal with that issue even from a distance. Using JS as a source code intentionally or unintentionally helps keeping JS platform independent.
And as a fan of WASM, I agree with you xDD
There is a lot of great stuff writen in JS you can't write in TS or it feels wrong. Personaly I using WASM as ascelerator, JS for libraries that are used a cross multiple projects, some up to 7 years old and TS for new apps. All that to meet you nead sort of glue.
Node Modules is the Universe
It was originally created by Zaphod Beeblebrox in 1988, but since there was no WWW yet, he programmed it to eventually truly answer The Question.
So Node Modules has to contain every F&@^ Byte all programmers in the world can come up with
Some say Node Modules is the ultimate practical joke.
There are too many dependencies for the dependencies that you are downloading, if these dependencies were updated to use modern JavaScript, you might find that JavaScript has a built-in feature that removes a lot of extra code that is being downloaded as dependent packages. It is the issue of libraries not using the full capabilities of a language as that language evolves -- this is true of Java, C#, etc... It would be a lot of work to rewrite to use those features and that is why old libraries are continually used. Python and Go have large standard libraries, that is why it is less of an issue in these languages. This is technical debt and it is everywhere in software development.
You can take a look at pnpm.io
I’ve got really interesting results in installation times and storage
Yes, I mentioned pnpm in my blog too. Its amazing!!
Recommending to try bun.sh as package manager. It is not solving problem of amount of space but speed up download process.
Yes, Bun has been a breath of fresh air in the JS ecosystem!!
Great! learned something new today. 🍵
That's great to hear!! :D
Dependencies in go and rust are source files, compiled with your application, and python or ruby dependencies are source files just like nodejs. Java dependencies are compiled and it is very heavy.
Thanks for the feedback!! I added some extra stuffs to explain it too. About Python you are correct, so removed it. Actually I didn't even noticed that I put it in the first place, but yeah the thing with large dependencies is not just limited to JS. About Java, the total dependencies size is usually smaller than the Node because maven and gradle is better with managing duplicate dependencies in the dependency tree, and they don't even need to download the source code. The duplication management is now finally starting to get handled by nice package managers such as pnpm
Thanks, It was very usefull
Thank You!!
Insightful!
Thank You!!