NPM Pack / Nodejitsu ignores buried Node_modules folders

Question

It seems that NPM pack, and by extension, nodejitsu, do not like my node_modules folders. :*(

I currently am building a web app.

My web app's project folder structure is as follows:

Root

Engine(folder)                      Server(folder)    readme.md     package.json

(Multiple Folders) (folder)(folder) node_modules(folder)

                   easyimage,mongodb,mysql(folders)       socket.io (folder)

       node_modules(folder, NPM Pack ignores this)           node_modules(folder, NPM Pack ignores this)

                                                       Socket.io-client (folder, NPM Pack ignores this)

I hope everyone can see this structure alright!

The problem I am having is that when I run NPM Pack at the root directory, the entire directory structure is packed correctly, except for all of the node_modules folders below the first node_modules folder.

It is as if NPM pack completely ignores those node_modules folders. (The one below socket.io for instance).

Due to the fact NPM pack ignores these npm folders, jitsu is also ignoring them and I can't get my web app started.

How can I get NPM pack/nodejitsu to correctly package all of the node_modules folders correctly?

My current package.json file, at the root directory, looks like follows: http://pastebin.com/SAU6rwb5

As you can see, I have attempted to use bundleDependencies to tell NPM Pack that I am trying to include some node_modules folders (modules?), but pack still ignores all of them... Also, if i include anything under "dependencies", NPM start creates a new (??) node_modules folder at the root directory... but at the root directory nothing needs node_modules... as you can see node_modules are used inside of the server folder.

How can I get NPM Pack to recognize the files and folders inside of all of the node_modules folders and pack them correctly?

Answer 1

(jump to last paragraph if you want a really easy solution)

I'm having a hard time understanding your app structure. I think I get what your trying to do though. From https://npmjs.org/doc/folders.html , it actually goes into detail about when and why sub modules will or wont show up.

When installing locally, npm first tries to find an appropriate prefix folder. This is so that npm install foo@1.2.3 will install to the sensible root of your package, even if you happen to have cded into some other folder.

Starting at the $PWD, npm will walk up the folder tree checking for a folder that contains either a package.json file, or a node_modules folder. If such a thing is found, then that is treated as the effective "current directory" for the purpose of running npm commands. (This behavior is inspired by and similar to git's .git-folder seeking logic when running git commands in a working dir.)

If no package root is found, then the current folder is used.

When you run npm install foo@1.2.3, then the package is loaded into the cache, and then unpacked into ./node_modules/foo. Then, any of foo's dependencies are similarly unpacked into ./node_modules/foo/node_modules/....

Bear With me for another quote...

Cycles are handled using the property of node's module system that it walks up the directories looking for node_modules folders. So, at every stage, if a package is already installed in an ancestor node_modules folder, then it is not installed at the current location.

Consider the case above, where foo -> bar -> baz. Imagine if, in addition to that, baz depended on bar, so you'd have: foo -> bar -> baz -> bar -> baz .... However, since the folder structure is: foo/node_modules/bar/node_modules/baz, there's no need to put another copy of bar into .../baz/node_modules, since when it calls require("bar"), it will get the copy that is installed in foo/node_modules/bar.

This shortcut is only used if the exact same version would be installed in multiple nested node_modules folders. It is still possible to have a/node_modules/b/node_modules/a if the two "a" packages are different versions. However, without repeating the exact same package multiple times, an infinite regress will always be prevented.

As far as the bundledDependencies:

Upon publishing [this also applies to packing], npm will look in the node_modules folder. If any of the items there are not in the bundledDependencies array, then they will not be included in the package tarball.

I take this to mean that it will only bundle the listed modules in ./node_modules/ and specific sub-modules for a ./package.json . Then of course as written above, it recursively walks down the directory tree again... so if it sees another package.json file in this directory npm will look and see if that has any bundled deps to include in the pack.

So as I understand it right now, since you have no packages in your base directory, your bundled dependencies in package.json doesn't do anything, and actually having items in your bundledDependencies field does more harm than good.

To fix you need to edit the package.json files to include these bundles at each level.

I have had this issue before when trying to get a packed then unpacked meteor app working on nodejitsu. I solved it in a different way. In the root folder of my app I included all of the top level node modules and explicitly set their version in my package.json file.

From what I understand your file structure is such:

    app
    +-- Engine
    +-- Server
    |   +-- socket.io
    |   |   `-- package.json
    |   |   +-- node_modules
    `-- readme.md
    `-- package.json

If this is so then you need to be editing the package.json under socket.io to include the bundled deps you want. Generally though you can trust the package maintainers to keep valid versions.(but in this case you can't?)

As for Socket.io-client not being packed that's a result of it being a dependency of socket.io.

If I was to suggest a way for you to make this easier for yourself, i'd suggest you to, in your MAIN top level package.json file, to include the dependencies you need for your app @ the specific version you need. If you need them bundled for some reason, add them into the bundled section, if you need sub modules at a different version than what the author intended. Consider making a folder called package or vendor and then placing the modules in there, in those you can edit the package.json and bundles their dependencies to your hearts content. Be sure though that you do not ignore any files or folders under you vendor or packages directory with .npmignore or .gitignore files.

Alternatively, if this is too difficult (editing all theses files and specifying certain versions can be a pain) I'd suggest hosting your vendor packages some where where you could download them with a script, and then execute this script in the postinstall part of your package.json (take a read at https://npmjs.org/doc/scripts.html ... you would add this in the same section you have your "start" script.

I hope that clarifies things.