[RRFC] npm run-series && npm run-parallel

[MylesBorins]

Motivation ("The Why")

Often an npm script is part of a more complicated pipeline, tools such as grunt + gulp have been created to help manage some of this pipeline. A repo may have multiple steps to a compilation such as clean, build various types of assets (html, css, js), etc. Currently npm itself does not have a mechanism to orchestrate more complex workflow.

Example

Most recently I utilized xargs to accomplish a similar task.

"build": "echo build:clean build:css build:js build:html build:img | xargs -n1 npm run"

if run-series were available this could be accomplished in a platform agnostic way via

"build": "npm run-series build:clean build:css build:js build:html build:img"

It could also be optimized to run parallel operations

"build": "npm run build:clean && npm run-parallel build:css build:js build:html build:img"

or potentially

"build": "npm run build:clean --then npm run-parallel build:css build:js build:html build:img"

How

Current Behaviour

Currently there is no way to accomplish this beyond using && or & which is can be platform specific

Desired Behaviour

First class support for running multiple scripts in parallel or series

References

• n/a

[rwjblue]

Currently there is no way to accomplish this beyond using && or & which is can be platform specific

@MylesBorins - Just to confirm, you are specifically talking about a built-in mechanism for this right? Is the proposal roughly that we should add a run-series command that works similarly to how npm-run-all (a somewhat popular package that provides some functionality in this space) works?

[ljharb]

for "run parallel", i haven't yet found a tool that properly handles nonzero exit codes, and also handles mixing (and prefixing) streaming stdout/stderr output. If npm can solve this that would be awesome.

[wesleytodd]

I think this is a real issue, and our current constraints have led to some great and some really awful things. If we make it more powerful we run the risk of encouraging some even more wild usage.

At work, we have a system for composing "commands" which is not make. Learning from this, it is hard to get right but is ultimately very useful. If we want to really solve this well, which I am not sure we need to do, then I think we should consider preexisting systems with features like work avoidance and DSLs for declaring the commands. Also, to share, we have a tool which uses the stmux package and is a pretty good DX for the parallel commands to solve the log-interleaving problem.

[MylesBorins]

FWIW I think that we could likely scope this solution to only run in series that is more or less sugar for npm run one && npm run two && npm run three && npm run four and significantly reduce duplication and noise in the package.json

[MylesBorins]

Removing Agenda label. A next step would be a making a more complete proposal specifically for series.

[Christian24]

As I said in the meeting I think there is also a use case I can see for parallel, although probably in a second RFC: In web development there are often watch tasks that run Typescript, Sass, webpack or whatever. It would be useful to have a way to run them in parallel through npm and if one dies, just kill the entire thing.

[ColtHands]

I can definitely find this useful, not even parallel, but running multiple scripts in series as an array:

"scripts": {
  "build": [ "webpack --verbose", "npm run test", "git add .", "echo 'all done'" ]
}

npm run build

I think run-parallel should be an addition to yet not implemented run-series

Will run-series be an easier implementation than run-parallel as npm can run one script and in theory it could run multiple?

[ljharb]

The array approach is a pretty straightforward way to implement "series", fwiw - I like it.

[wesleytodd]

Not that I like this line of thought, but to toss out another complicated option:

• Array: run in series
• Object: run in parallel where the names are headings for the grouped logs

You could (although I strongly believe we need to move this all outside of npm) even allow for nesting:

"scriptfoo": [{
  "build1": "thing",
  "build2": "otherthing"
}, "test"]

Again, I really dont like the complexity this would add to npm, but if someone were to work on a declarative and simple command runner, one might do it this way.

[muuvmuuv]

Uhh, I like this solution a lot @wesleytodd because it does not introduce commands/flags and is straight forward! Through a flag like --no-fail we could say that parallel or series tasks wouldn't exit on error.

Our scripts are looking like that with npm-run-all:

  "scripts": {
    "start": "ng serve",
    "prod": "ng serve --prod",
    "clean": "del dist tmp out-tsc coverage results abstracts .lighthouseci",
    "proxy": "make restart",
    "build": "run-s build:*",
    "build:ng": "ng build --prod",
    "build:min": "node scripts/minify-files.js",
    "docs": "typedoc --options .typedoc.js src",
    "docs-open": "open-cli docs/index.html",
    "all": "run-s clean lint build docs test-ci e2e-ci perf",
    "−−−−−−−−−−−−−−−---−−−---- TESTS −---−−−−−−---−−−−−−−−−−−": "",
    "test": "ng test",
    "test-ci": "ng test -c ci",
    "e2e": "ng e2e",
    "e2e-ci": "ng e2e -c ci",
    "−−−−−−−−−−−−−−−---−−−- PERFORMANCE −−−−−−−---−−−−−−−−−−−": "",
    "perf": "run-s build proxy perf-lhci perf-open",
    "perf-lhci": "lhci collect",
    "perf-open": "run-p perf-open:*",
    "perf-open:lhci": "lhci open",
    "−−−−−−−−−−−−−−−−−−------ LINTING −−−−−−−------−−−−−−−−−−": "",
    "lint": "run-p lint:*",
    "lint:tsc": "tsc --build tsconfig.app.json",
    "lint:tsc-worker": "tsc --build tsconfig.worker.json",
    "lint:source": "ng lint",
    "lint:cypress": "eslint --ext .ts,.js cypress",
    "lint:engine": "eslint --cache *.js scripts/**/*.js",
    "lint:scss": "stylelint --cache src/**/*.scss",
    "−-−−−−−−−−−−−−−−−-----  FORMATING  −−−−−−--−−−−−−−−−−-−−": "",
    "format": "run-p format:*",
    "format:typings": "prettier --write 'src/**/*.d.ts'",
    "−-−−−−−−−−−−−−−−−-−----−  TOOLS  −−−−−−----−−−−−−−−−−-−−": "",
    "preinstall": "node scripts/welcome.js && sh scripts/nvm-use.sh",
    "postinstall": "node scripts/setup.js",
    "generate-graphql": "graphql-codegen && npm run format:typings",
    "changelog": "conventional-changelog -p angular -i CHANGELOG.md -s -r 0",
    "release": "node scripts/release.js"
  },

This is how it could look like then (sub-tasks would be indented with there title):

{
  "scripts": {
    // "start": "ng serve",
    // "prod": "ng serve --prod",
    // "clean": "del dist tmp out-tsc coverage results abstracts .lighthouseci",
    // "proxy": "make restart",
    // "docs": "typedoc --options .typedoc.js src",
    // "docs-open": "open-cli docs/index.html",
    "all": [
      "clean",
      "lint",
      "build-docs",
      "test-ci",
      "e2e-ci",
      "perf"
    ],
    "−−−−−−−−−−−−−−−---−−−---- BUILD −---−−−−−−---−−−−−−−−−−−": "",
    "build": {
      "Build application": "build-ng",
      "Minify non-minified": "build-min"
    },
    "build-ng": "ng build --prod",
    "build-min": "node scripts/minify-files.js",
    // "−−−−−−−−−−−−−−−---−−−---- TESTS −---−−−−−−---−−−−−−−−−−−": "",
    // "test": "ng test",
    // "test-ci": "ng test -c ci",
    // "e2e": "ng e2e",
    // "e2e-ci": "ng e2e -c ci",
    "−−−−−−−−−−−−−−−---−−−- PERFORMANCE −−−−−−−---−−−−−−−−−−−": "",
    "perf": {
      "Building": "build",
      "Starting proxy": "proxy",
      "Running performance tests": "perf:*",
      "Opening reports": "perf-open"
    },
    "perf:lhci": "lhci collect",
    "perf-open": "perf-open:*",
    "perf-open:lhci": "lhci open",
    "−−−−−−−−−−−−−−−−−−------ LINTING −−−−−−−------−−−−−−−−−−": "",
    "lint": "lint:*",
    "lint:tsc": "tsc --build tsconfig.app.json",
    "lint:tsc-worker": "tsc --build tsconfig.worker.json",
    "lint:source": "ng lint",
    "lint:cypress": "eslint --ext .ts,.js cypress",
    "lint:engine": "eslint --cache *.js scripts/**/*.js",
    "lint:scss": "stylelint --cache src/**/*.scss",
    "−-−−−−−−−−−−−−−−−-----  FORMATING  −−−−−−--−−−−−−−−−−-−−": "",
    "format": "format:*",
    "format:typings": "prettier --write 'src/**/*.d.ts'",
    // "−-−−−−−−−−−−−−−−−-−----−  TOOLS  −−−−−−----−−−−−−−−−−-−−": "",
    // "preinstall": "node scripts/welcome.js && sh scripts/nvm-use.sh",
    // "postinstall": "node scripts/setup.js",
    // "generate-graphql": "graphql-codegen && npm run format:typings",
    // "changelog": "conventional-changelog -p angular -i CHANGELOG.md -s -r 0",
    // "release": "node scripts/release.js"
  }
}

* would just serve as a glob like npm-run-all handles it :)

[guybedford]

Perhaps the array should always be the names of other scripts in the script map? This way the individual tasks remain named scripts that can be run separately. In addition it removes the need to inline npm run:

{
  "scripts": {
    "build-all": ["npm run build1", "npm run build2"],
    "build1": "...",
    "build2": "..."
  }
}

and instead just write:

{
  "scripts": {
    "build-all": ["build1", "build2"],
    "build1": "...",
    "build2": "..."
  }
}