Offline buildpack still downloading files

[ArthurHlt]

I'm currently using the offline buildpack on my own cloudfoundry instance and i ran npm install before pushing my app to have all my dependencies already in the app. But when the app staging the compilation run npm rebuild and this command go to download nodejs source (i don't know why) from internet which is pretty annoying when you are offline :)
I know that the compile script come from heroku but Is there a way to make it not download nodejs source?

Recreate the issue by following this instruction for example:

1. Download https://github.com/cloudfoundry-community/etherpad-lite-cf/releases/download/1.4.0-cf/etherpad-lite-cf.zip (this is a nodejs app which already set for being offline)
2. Extract the archive
3. go to extracted archive
4. push the app
5. In log we can see http get from npm

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this. You can view the current status of your issue at: https://www.pivotaltracker.com/story/show/78320918.

[azzamallow]

Hi @ArthurHlt, thank-you for submitting this issue.

I attempted to recreate the problem, I did not see errors relating to http get and the app starts up fine. Are you able to provide the logs which shows the problem?

[ArthurHlt]

Hi @azzamallow ,
First logs with my proxy set in env var for cloudfoundry (i don't want use it anymore):

...
-----> Requested node range:  >=0.6.3
-----> Resolved node version: 0.10.28
-----> Using node package from dependencies directory
-----> Found existing node_modules directory; skipping cache
-----> Rebuilding any native dependencies
       > [email protected] install /tmp/staged/app/node_modules/ueberDB/node_modules/pg
       > node-gyp rebuild || (exit 0)
       gyp http GET http://nodejs.org/dist/v0.10.28/node-v0.10.28.tar.gz
       gyp http 200 http://nodejs.org/dist/v0.10.28/node-v0.10.28.tar.gz
       gyp http GET http://nodejs.org/dist/v0.10.28/SHASUMS.txt
       gyp http 200 http://nodejs.org/dist/v0.10.28/SHASUMS.txt
...

Second, when i remove my proxy:

-----> Resolved node version: 0.10.28
-----> Using node package from dependencies directory
-----> Found existing node_modules directory; skipping cache
-----> Rebuilding any native dependencies
       > [email protected] install /tmp/staged/app/node_modules/ueberDB/node_modules/pg
       > node-gyp rebuild || (exit 0)
       gyp http GET http://nodejs.org/dist/v0.10.28/node-v0.10.28.tar.gz
       gyp WARN install got an error, rolling back install
       gyp ERR! configure error
       gyp ERR! stack Error: getaddrinfo EADDRINFO
       gyp ERR! stack     at errnoException (dns.js:37:11)
       gyp ERR! stack     at Object.onanswer [as oncomplete] (dns.js:124:16)
       gyp ERR! System Linux 3.0.0-32-virtual
       gyp ERR! command "node" "/tmp/staged/app/vendor/node/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
       gyp ERR! cwd /tmp/staged/app/node_modules/ueberDB/node_modules/pg
       gyp ERR! node -v v0.10.28
       gyp ERR! node-gyp -v v0.13.0
       gyp ERR! not ok

[squeedee]

How did you clear/remove the proxy var?

[squeedee]

My other concern, this was an issue with the Node buildpack some time back. Have you tried the latest buildpack: https://github.com/cloudfoundry/nodejs-buildpack/releases/tag/v1.0.1

[ArthurHlt]

I don't use cf unset-env i prefer delete the app and recreate it. Well, i have it own my cloudfoundry but the problem is not here cause even if download is removed from npm it will fail cause of this line: https://github.com/cloudfoundry/nodejs-buildpack/blob/master/bin/compile#L74 :/

[jchesterpivotal]

@ArthurHlt I also took a quick look and I can't recreate this either. Can you tell us a bit more about your proxy variables?

[lyuyun]

I got the same problem when push my node app with Node Buildpack v1.0.1, my logs:

...
       > node-gyp configure build
       gyp info it worked if it ends with ok
       gyp verb cli [ 'node',
       gyp verb cli   '/tmp/staged/app/vendor/node/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js',
       gyp verb cli   'configure',
       gyp verb cli   'build' ]
       gyp info using [email protected]
       gyp info using [email protected] | linux | x64
       gyp verb command configure []
       gyp verb check python checking for Python executable "python" in the PATH
       gyp verb `which` succeeded python /usr/bin/python
       gyp verb check python version `python -c "import platform; print(platform.python_version());"` returned: "2.6.5\n"
       gyp verb get node dir no --target version specified, falling back to host node version: v0.10.29
       gyp verb command install [ 'v0.10.29' ]
       gyp verb install input version string "v0.10.29"
       gyp verb install installing version: 0.10.29
       gyp verb install --ensure was passed, so won't reinstall if already installed
       gyp verb install version not already installed, continuing with install 0.10.29
       gyp verb ensuring nodedir is created /home/vcap/.node-gyp/0.10.29
       gyp verb created nodedir /home/vcap/.node-gyp
       gyp http GET http://nodejs.org/dist/v0.10.29/node-v0.10.29.tar.gz
       gyp verb download using proxy url: "http://9.91.13.41:8081"
       gyp http 200 http://nodejs.org/dist/v0.10.29/node-v0.10.29.tar.gz
       gyp verb extracted file from tarball deps/uv/src/win/atomicops-inl.h
       gyp verb extracted file from tarball deps/uv/src/win/internal.h
       gyp verb extracted file from tarball deps/uv/src/win/winapi.h
       gyp verb extracted file from tarball deps/uv/src/win/winsock.h
       gyp verb extracted file from tarball deps/uv/src/win/stream-inl.h
...

[jchesterpivotal]

@lyuyun Do you have an app we can look at? We still haven't managed to recreate this fault.

[lyuyun]

@jchesterpivotal I tried to push the nodebb app, and add "-dd" param in bin/compile to get more dump info when npm install.

[jtarchie]

We were able to recreate the issue in our local testing environment with an offline CF. It turns that npm and node-gyp are not well suited for an offline environment.

When it comes to compiling dependencies we cannot guarantee that they will not make calls to the Internet. The offline-buildpack is only able to make sure that nodejs and npm are installed correctly in an offline environment.

We'd have to make internal changes to npm to support a true offline buildpack, which at this time is beyond the scope of the buildpack team. The topic of extending npm was discuss here.