This article covers an old version of pnpm. For an updated version of the article go here.
New users of pnpm frequently ask me about the weird structure of node_modules that pnpm creates. Why is it not flat? Where are all the sub-dependencies?
I am going to assume that readers of the article are already familiar with flat
node_modulescreated by npm and Yarn. If you don't understand why npm 3 had to start using flatnode_modulesin v3, you can find some prehistory in Why should we use pnpm?.
So why is pnpm's node_modules unusual? Let's create two directories and run npm install express in one of them and pnpm install express in the other one. Here's the top of what you get in the first directory's node_modules:
You can see the whole directory here.
And this is what you get in the node_modules created by pnpm:
You can check it here.
So where are all the dependencies? There is only one folder in the node_modules called .registry.npmjs.org and a symlink called express. Well, we installed only express, so that is the only package that your application has to have access to
Read more about why pnpm's strictness is a good thing here
Let's see what is inside express:
express has no node_modules? Where are all the dependencies of express?
The trick is that express is just a symlink. When Node.js resolves dependencies, it uses their real locations, so it does not preserve symlinks. But where is the real location of express, you might ask?
Here: node_modules/.registry.npmjs.org/express/4.16.3/node_modules/express.
OK, so now we know the purpose of the .registry.npmjs.org/ folder. .registry.npmjs.org/ stores all the packages in a flat folder structure, so every package can be found in a folder named by this pattern:
.registry.npmjs.org/<name>/<version>/node_modules/<name>
This flat structure avoids the long path issues that were caused by the nested node_modules created by npm v2 but keeps packages isolated unlike the flat node_modules created by npm v3,4,5,6.
Now let's look into the real location of express:
Is it a scam? It still lacks node_modules! The second trick of pnpm's node_modules structure is that the dependencies of packages are on the same directory level on which the real location of the dependent package. So dependencies of express are not in /express/4.16.4/node_modules/express/node_modules/ but in /express/4.16.4/node_modules/:
All the dependencies of express are symlinks to appropriate directories in node_modules/.registry.npmjs.org/. Placing dependencies of express one level up allows avoiding circular symlinks.
So as you can see, even though pnpm's node_modules structure seems unusual at first
- it is completely Node.js compatible
- packages are nicely grouped with their dependencies
The structure is a little bit more complex for packages with peer dependencies but the idea is the same: using symlinks to create a nesting with a flat directory structure.
If you'd like to try out pnpm, you can easily install it with npm: npm i -g pnpm. Then just run it instead of npm when you need to install something: pnpm install foo bar.
Top comments (9)
Hey Zoltan, I had not heard of pnpm before but installed it now because I really like the underlying (engineering) principles you mention here. I see it also support workspaces, so I will try and see if I can apply it in my Stardazed project which is a quickly-growing monorepo. It mostly uses peerDependencies, though.
I'll also look into what Jan mentioned about the recursive commands and see what they are.
My main initial feedback is that the name pnpm can be confusing to some as it implies some relation between the product and the npm org. It's probably too late to change this now but I can see why FB went with the name "yarn" as it is clearly something else.
Besides that though, looks good! Thanks!
Updated to add:
I already had the first benefit of using this as I have devDeps that work in node (rollup) and it installs the @types/node package which was causing VS Code to add a ton of Node types in the autocomplete for my browser-based package I'm writing. Now that those packages are whiffed away, I don't have to work around this anymore! Nice.
2nd update:
Ahh,
pnpm recursive run ...could indeed make build processes a lot less script-heavy. Great stuff!Hey Arthur! Let us know if you have any issues with pnpm or questions.
pnpm (performant npm) was a name given by the initial author of pnpm - Rico Sta. Cruz. It was based on ideas of ied - another js package manager with a bad name😄.
We are aware of the bad name issue and maybe we'll rename it in the future. At the right time.
Thanks Zoltan, I figured out quickly indeed that the recursive commands make working with monorepos a lot easier. Very nice. I was looking through your docs and was wondering what the purpose is of
pnpm server. Is it for CI contexts?The server feature was added for glitch. Their use case is that they have many containers that use a shared store but the containers don't have access to the store. So when an installation is done in a container, actions like "download a tarball from the registry" or "link a package from the store to my project" are delegated to the store server.
It probably deserves a separate article.
I see big potential in the store server. Like in the future it could keep the store warm or seed the packages in a p2p network.
Got it, thanks for explaining again. I'll convert some projects and file any issues I may find.
We (the maintainers of Cycle.js) are currently migrating from yarn to pnpm. Our main motivation:
recursivecommands (especiallyrecursive linkfor using our examples as end-2-end tests)I am currently redoing our complete tooling and it allows us to throw away our Makefile and a lot of custom ad-hoc scripts.
Thanks for writing such a good piece of tooling!
Awesome! I really like Cycle.js!
If you have any issues or need some features, feel free to write in our chatroom or create new issues in the repo.
Is
pnpmcompatible withlerna?I don't know, I did not try it but I guess you could use lerna for publishing and
pnpm recursivecommands for everything else