Trim your node_modules repetition by over 60%
At Illyriad Games we run a tight ship with low dependencies and we need to know the costs, risks, advantages and disadvantages of everything we include and use. Being a very small studio, this is super-important.
This doesn’t mean not using the right tools for the job, or rejecting things because they weren’t invented here: it means the exact opposite. Even our projects are a mix of different types as each one is suited for a different need (see image).
Indeed, we like open source as it gives us the best of both worlds: a great diversity of minds from all around the world with wider viewpoints then our own – including specialists in their field – and with the ability to contribute if we need changes or fixes to suit our more particular needs. Generally the community will improve and build on our contribution; enriching it for better code for all. Much like life; diversity is a good thing!
It also allows us to concentrate on what we are good at: mechanical sympathy and going really really fast. When going fast, the most important thing is to reduce unnecessary work. The less work you need to do per task, the more tasks you can do. This is both as a person and for the computer. Doing less also scales better:
@robmiles True, doing nothing scales infinitely! Webscale, even! ;)— Scott Hanselman (@shanselman) January 18, 2012
This is one of the reasons we are quite excited by .NET Core and ASP.NET 5. They are both open source as well as being leaner and more modular – meaning we only need to include what we use. It’s also cross platform and has all sorts of other goodness.
So, in that spirit, today I’m looking at our build process and seeing if there is any way that it can be trimmed.
We currently use Gulp for the task, though you can use Grunt and probably other things if they suit your needs better.
Our list of modules is probably quite slim already, at 20:
Looking in the node_modules directory:
That’s only 24 folders; but how many is that really?
So the 20 modules we use require 62MB over 15k files and 4.2k folders to work. Seems quite high; some of this is that modules that use other modules have their own node_modules directories, and they are included recursively.
This has the added complication of making very long file paths – which causes problems even when trying to delete them:
Is there anything we can do to reduce this? Turns out there is.
Just in case… Make sure you back up your project!
Upgrade to npm 3+ (currently v3.2.2) as this has a new feature where it tries to install modules and dependancies in as flat a structure as possible:
Flat, flat, flat!
Your dependencies will now be installed maximally flat. Insofar as it is possible, all of your dependencies (and their dependencies, and THEIR dependencies) will be installed in your project’s node_modules folder with no nesting.
Bear in mind npm 3 is currently in beta. So you may find some issues using it.
To upgrade npm in windows we used npm-windows-upgrade; instructions are included. But basically: in an elevated powershell prompt run the following:
Now your npm version should be the most recent.
Next, upgrade all version numbers to most recent versions in your project’s package.json. Visual Studio’s editor makes this pretty easy; delete the current version number and it’s surrounding quotes, open a double quote and it will give you a drop down with a list of the recent versions to choose from. Save and close the project.
Next step is deleting the current
node_modules. This isn’t as straightforward due to the exceptionally long paths involved, as outlined above. Windows explorer won’t work, nor will
rd at command line. However, you can do it using the built in
robocopy: Robust File Copy for Windows. The process is a bit weird 😄
If you ask it to copy files from a source to a destination and purge anything there that doesn’t match, it will delete all the files. At a command prompt go into your project directory which has the
node_modules folder, do a purge copy, then remove the directory which will now have the files from your top level folder in it; as follows:
Now you can open your project in Visual Studio 2015, wait for it to do its package restore magic; not sure of the exact steps for this (if its auto-magic on project open, or open and edit package.json, then save it).
You can follow the progress in the Output window in the
Bower/NPM section. Once this has finished, close the project.
Next, run a dedupe step from the command prompt in your project root (the directory that contains node_modules); just as a double check:
Ok, let’s see what that has done to the
Whoa, it’s exploded! There are now 280 folders in there rather than 24 – that’s a 1167% increase. Or so it seems… Let’s look at what it really means:
26MB over 5.7k files and 1.5k folders. That means disk space is down by 60%; file count is down by 62%; and folder count is down by 64%. So actually its a pretty good improvement!
Not everything has fully deduped, as modules will still nest underneath one another when two (or more) modules have conflicting dependencies. Looking at
browserify, for example, it has the following nested modules:
Which makes sense as its due to a version conflict with our directly included
through2 as its a full version bump.
I doubt this particularly speeds up build times, or that the extra drive space is hugely important, but it does make the completionist in me happy; as well as reducing the max path to only 192 characters rather than over 290 characters which makes parts of Windows happy. Although the .NET team are already looking into this issue:
More importantly, with this comes greater knowledge of module dependencies, risks and costs.
For the convenience this gives us for our build steps, this is a price we are more than happy to pay. However, if it was a runtime cost because we were running a node.js or io.js server; it would certainly be something I’d want to look at more closely!
Also understanding your true dependencies can be very important from a security standpoint.
Discuss this in our forums: Forum Discussion: Reduce node_modules recursion and long paths for ASP.NET 5