Yarn install skips plugins during deployment

[gsaks123]

During deployment, Yarn install seems to be installing only a partial/incomplete list of dependencies into my node_modules folder.

If I manually SSH into the server and go to the release folder for the deployment that just failed, I can run yarn and it will properly install all necessary dependencies. It is only during Capistrano deployment when yarn will skip necessary dependencies, as if it is referencing a generic list.

Furthermore, I tried the approach outlined here:
#1037 (comment)

This actually will work properly, but then gets over-written by bundle exec rake assets:precompile. First it will run the custom task, and next it will run the assets:precompile. As I investigated, I made the deployment fail by forcing an error into the first script:

namespace :deploy do
  desc 'Run rake yarn:install'
  task :yarn_install do
    on roles(:web) do
      within release_path do
        execute("cd #{release_path} && yarn install")
        false + 1 ### force an error to make the deployment fail
      end
    end
  end
end

At this point if I SSH into the server and look at my node_modules folder, all the necessary dependencies have been installed from the custom task. It is only once bundle exec rake assets:precompile runs where yarn install runs again and does not include all of the necessary dependencies.

I experimented with adding this to my environment file but it did not seem to matter:
config.webpacker.check_yarn_integrity = false

Application / Environment info:

• Rails 4.2.10
• Ruby 2.2.2
• Node 9.11.1
• Yarn 1.6.0

I don't see very much Rails 4 discussion here so I don't know if that is going to be a problem.

.babelrc contents:

{
  "env": {
    "test": {
      "plugins": ["transform-es2015-modules-commonjs"]
    }
  },
  "presets": [
    ["env", {
      "modules": false,
      "targets": {
        "browsers": "> 1%",
        "uglify": true
      },
      "useBuiltIns": true
    }]
  ],
  "plugins": [
    ["module-resolver", {"root": ["./app/assets/javascript/src"]}],
    "syntax-dynamic-import",
    "transform-object-rest-spread",
    ["transform-class-properties", { "spec": true }]
  ]
}

package.json contents:

{
  "name": "",
  "description": "",
  "repository": "",
  "author": "",
  "license": "UNLICENSED",
  "private": true,
  "engines": {
    "node": "9.11.1"
  },
  "dependencies": {
    "@rails/webpacker": "^3.4.3",
    "babel-polyfill": "^6.26.0",
    "jquery": "^3.3.1",
    "jquery-migrate": "3.0.0",
    "jquery-textfill": "^0.6.0",
    "jquery-ui-dist": "^1.12.1",
    "jquery-ujs": "^1.2.2",
    "jquery.scrollto": "^2.1.2",
    "rickshaw": "^1.6.5",
    "sticky-kit": "^1.1.3"
  },
  "devDependencies": {
    "babel-eslint": "^8.2.3",
    "babel-jest": "^22.4.3",
    "babel-plugin-module-resolver": "^3.1.1",
    "babel-plugin-transform-es2015-modules-commonjs": "^6.26.0",
    "babel-preset-env": "^1.6.1",
    "eslint": "^4.19.1",
    "eslint-plugin-jest": "^21.15.0",
    "jest": "^22.4.3",
    "webpack-dev-server": "^3.1.3",
    "yarn": "^1.6.0"
  },
  "scripts": {
    "test": "jest --watch"
  }
}

The deployment output:

      01 bundle exec rake assets:precompile
      01 yarn install v1.6.0
      01 [1/5] Validating package.json...
      01 [2/5] Resolving packages...
      01 [3/5] Fetching packages...
      01 info [email protected]: The platform "linux" is incompatible with this module.
      01 info "[email protected]" is an optional dependency and failed compatibility check. Excluding it from installation.
      01 [4/5] Linking dependencies...
      01 warning "@rails/webpacker > [email protected]" has unmet peer dependency "caniuse-lite@^1.0.30000697".
      01 warning " > [email protected]" has unmet peer dependency "babel-core@^6.0.0 || ^7.0.0-0".
      01 warning " > [email protected]" has unmet peer dependency "webpack@^4.0.0-beta.1".
      01 warning "webpack-dev-server > [email protected]" has unmet peer dependency "webpack@^4.0.0".
      01 [5/5] Building fresh packages...
      01 Done in 6.55s.
      01 Webpacker is installed 🎉 🍰
      01
      01 Using /home/deployer/tapology/releases/20180421190039/config/webpacker.yml file for setting up webpack paths
      01
      01 RAILS_ENV=staging environment is not defined in config/webpacker.yml, falling back to production environment
      01 Compiling…
      01 Compilation failed:
      01 Hash: 4fcb9af549025ddd52e5
      01 Version: webpack 3.11.0
      01 Time: 1013ms
      01                                        Asset       Size  Chunks             Chunk Names
      01      application-5e0d299f19da31c514c5.js.map    1.02 kB       4  [emitted]  application
      01     userWriteupsEdit-5e9d17bf56500316e3a7.js    2.12 kB       0  [emitted]  userWriteupsEdit
      01           ownerships-d93d3fe159498675edf0.js    2.11 kB       2  [emitted]  ownerships
      01         fightersShow-43314617ed34c387dbad.js    2.11 kB       3  [emitted]  fightersShow
      01          application-5e0d299f19da31c514c5.js    2.11 kB       4  [emitted]  application
      01                admin-58a40ea0471fe9339953.js     2.1 kB       5  [emitted]  admin
      01 userWriteupsEdit-5e9d17bf56500316e3a7.js.map    1.02 kB       0  [emitted]  userWriteupsEdit
      01            posts-c4c633e642216e15f1d6.js.map    1.01 kB       1  [emitted]  posts
      01       ownerships-d93d3fe159498675edf0.js.map    1.02 kB       2  [emitted]  ownerships
      01     fightersShow-43314617ed34c387dbad.js.map    1.02 kB       3  [emitted]  fightersShow
      01                posts-c4c633e642216e15f1d6.js     2.1 kB       1  [emitted]  posts
      01            admin-58a40ea0471fe9339953.js.map    1.01 kB       5  [emitted]  admin
      01                                manifest.json  814 bytes          [emitted]
      01  userWriteupsEdit-5e9d17bf56500316e3a7.js.gz  782 bytes          [emitted]
      01             posts-c4c633e642216e15f1d6.js.gz  772 bytes          [emitted]
      01        ownerships-d93d3fe159498675edf0.js.gz  778 bytes          [emitted]
      01      fightersShow-43314617ed34c387dbad.js.gz  780 bytes          [emitted]
      01       application-5e0d299f19da31c514c5.js.gz  776 bytes          [emitted]
      01             admin-58a40ea0471fe9339953.js.gz  773 bytes          [emitted]
      01                             manifest.json.gz  278 bytes          [emitted]
      01    [0] ./app/assets/javascript/packs/admin.js 1.57 kB {5} [built] [failed] [1 error]
      01    [1] ./app/assets/javascript/packs/application.js 1.57 kB {4} [built] [failed] [1 error]
      01    [2] ./app/assets/javascript/packs/fightersShow.js 1.57 kB {3} [built] [failed] [1 error]
      01    [3] ./app/assets/javascript/packs/ownerships.js 1.57 kB {2} [built] [failed] [1 error]
      01    [4] ./app/assets/javascript/packs/posts.js 1.57 kB {1} [built] [failed] [1 error]
      01    [5] ./app/assets/javascript/packs/userWriteupsEdit.js 1.57 kB {0} [built] [failed] [1 error]
      01
      01 ERROR in ./app/assets/javascript/packs/application.js
      01 Module build failed: ReferenceError: Unknown plugin "module-resolver" specified in "/home/deployer/tapology/releases/20180421190039/.babelrc" at 0, attempted to resolve relative to "/home/deployer/tapology/releases/20180421190039"
      01     at /home/deployer/tapology/shared/node_modules/babel-core/lib/transformation/file/options/option-manager.js:180:17
      01     at Array.map (<anonymous>)
      01     at Function.normalisePlugins (/home/deployer/tapology/shared/node_modules/babel-core/lib/transformation/file/options/option-manager.js:158:20)
      01     at OptionManager.mergeOptions (/home/deployer/tapology/shared/node_modules/babel-core/lib/transformation/file/options/option-manager.js:234:36)
      01     at OptionManager.init (/home/deployer/tapology/shared/node_modules/babel-core/lib/transformation/file/options/option-manager.js:368:12)
      01     at File.initOptions (/home/deployer/tapology/shared/node_modules/babel-core/lib/transformation/file/index.js:212:65)
      01     at new File (/home/deployer/tapology/shared/node_modules/babel-core/lib/transformation/file/index.js:135:24)
      01     at Pipeline.transform (/home/deployer/tapology/shared/node_modules/babel-core/lib/transformation/pipeline.js:46:16)
      01     at transpile (/home/deployer/tapology/shared/node_modules/babel-loader/lib/index.js:50:20)
      01     at /home/deployer/tapology/shared/node_modules/babel-loader/lib/fs-cache.js:118:18
      01     at ReadFileContext.callback (/home/deployer/tapology/shared/node_modules/babel-loader/lib/fs-cache.js:31:21)
      01     at FSReqWrap.readFileAfterOpen [as oncomplete] (fs.js:434:13)
      01

[gsaks123]

I realize I missed this issue which led me to a temporary solution:
#1330

However, instead of modifying the Rake task to call yarn install with different options, I just modified it to completely skip yarn install:

# lib/tasks/webpacker.rake
Rake::Task["webpacker:yarn_install"].clear

namespace :webpacker do
  desc "Skip default webpacker yarn install"
  task :yarn_install do
    puts "Skipping webpacker yarn install"
  end
end

And then if I use this to run Yarn install manually during deploy, prior to asset precompilation, everything seems to succeed.

[somebody32]

I think the problem is that you have babel plugins in devDependencies and yarn is not installing them when NODE_ENV is set to production. Try to move everything related to build to dependencies

[chadwilken]

@somebody32 that obviously will work, but in old versions you could separate your dev and general deps

[somebody32]

@chadwilken not sure what you mean by "old versions", that is how yarn behaves, not webpacker here

[chadwilken]

@somebody32 I know that's what yarn is supposed to do. I didn't see he was on Rails 4.x. In older versions of Rails (5.1.x) w/ Webpacker they had a task in Rails for yarn install that installed all dependencies.

[gauravtiwari]

@chadwilken Yeah the rake task is updated with production flag which breaks the default behavior. I think we should remove that and let yarn use NODE_ENV to figure that out.

// cc @guilleiguaran

[a-barbieri]

I ended up following @somebody32 suggestion (Ruby on Rails 5.2.1, Webpacker 3.5.5).

Try to move everything related to build to dependencies

A thing that I found out is that it doesn't work if you don't remove yarn.lock and package-lock.json files from both local and remote server BEFORE precompiling your assets.

Here's the sequence I followed:

1. remove yarn.lock and package-lock.json from local machine and from remote server.
2. on your local machine in package.json move"babel-plugin-module-resolver": "^3.1.1" from devDependencies to dependencies.
3. on your local machine run yarn install.
4. commit the new package.json, yarn.lock and package-lock.json files
5. deploy them to your remote server
6. on your remote server run yarn install (not RAILS_ENV=production rails yarn:install) BEFORE running RAILS_ENV=production rails assets:precompile.

[a-barbieri]

Step 6 should be done by assets:precompile which should already taking care of yarn install, but I couldn't make it work.

Maybe that's due to the fact I was prepending RAILS_ENV=production? Dunno.

Anyway I got to that conclusion after reading @gauravtiwari comment here.