Smoke testing with select npm packages

[rvagg]

This discussion came up on the TC meeting today, prompted by a question on IRC, noting it here as a TODO if someone has spare energy and time to devote to starting this effort.

It would be ideal if io.js was regularly tested against a list of npm packages to test for breakage. Perhaps the list could comprise some of the most popular packages and/or some of the most interesting use-cases of Node/io.js to test for edge-cases. The tests could be simply running the test suites of specific versions against the given version of io.js.

[therebelrobot]

I would love to help out, though I'm not sure where all this would be done... a digitalocean instance maybe? Or could we set up a task in travisCI?

[chrisdickinson]

If I can feed the static analysis output into a queryable database, we should be able to get a list of "affected modules" given a changelist fairly cheaply. If we further filter that against a minimum number of downloads or dependents, we can at least get automated smoke tests.

I'd love to help out with this effort.

[rvagg]

@therebelrobot open question how it actually gets done but I suspect we'd be best to use our own CI resources, so at the moment it might look like a combination of Jenkins and build servers somewhere. DigitalOcean and/or Rackspace for Linux, Rackspace for possible Windows testing (might be best to defer that one!).

[rvagg]

oh, if anybody wants to make a start on this then I can provide server resources on one of our free accounts, if it's legitimately for this effort and won't be wasted resource.

[therebelrobot]

That makes sense. I would love to take a look at it, though I may not be the best to take point on it, given how much experience I have on automated regression testing (which is little).

[jbergstroem]

@rvagg how about a new job in jenkins we run as part of the RC/release procedure? We could check in a list packages (or have it as part of documentation that we can parse somehow) that we pass to the build and run. npm install would return non-zero exit codes so we'd at least get failures as a start. Adding more functionality to make output more visible could be a next step.

[Fishrock123]

Subbing to this issue, the big question I have is: is there any way we could know / try to run any arbitrary module's tests?

I suppose we could try a bunch of the most common options?

[ceejbot]

Pre-select a group of modules, top 50 downloaded from npm to start, and choose only modules where npm install && npm test works to run the tests. You could seed the VM with a few common resources (figure out what the modules need by looking at their .travis.yml files) or to simplify, start with modules that need nothing.

[jbergstroem]

I think selecting 10 packages – pure js and native – that doesn't have dependencies such as mysql and lives in the top 100 would be a good enough start.

Edit: ..with the added benefit of being able to trust their test suites enough to have it part of our verification.

[rvagg]

I started work on this here: https://github.com/rvagg/iojs-smoke-tests

Uses Docker to isolate so it's limited to Linux atm. The procedure could also work on a dedicated machine too so perhaps if we wanted to multi-OS this we could. For now though, the tricky bit is in making the test runs informative and non-flakey.

[mhdawson]

We already do this in our internal (IBM) builds for a list of modules and @jasnell was working to pull this over into the Node builds. It was originally in perl so he was going to port it over to run under Node. Right now we run in 2 flavors. 1) just do the install, 2) do the install and run built in tests for the module. The second is a subset of the first as its not always easy to automatically identify and run the built in tests. One thought we had along the way would be if there was guidance from the Node community to module developers with respect to how to include tests so that they could be easily run.

More recently we've also broken it out into 2 different types of runs. One which fixes the module versions so that we can tell if its the runtime that causes regressions. A second runs with the latest versions of the modules but a fixed runtime level so that we can identify regressions in the modules themselves.

[jasnell]

The node port on this is still in plan, it just ended up being pushed to a
lower priority for me. Can push it up the stack if need be.
On Jun 1, 2015 6:26 AM, "Michael Dawson" notifications@github.com wrote:

We already do this in our internal (IBM) builds for a list of modules and
@jasnell https://github.com/jasnell was working to pull this over into
the Node builds. It was originally in perl so he was going to port it over
to run under Node. Right now we run in 2 flavors. 1) just do the install,
2) do the install and run built in tests for the module. The second is a
subset of the first as its not always easy to automatically identify and
run the built in tests. One thought we had along the way would be if there
was guidance from the Node community to module developers with respect to
how to include tests so that they could be easily run.

More recently we've also broken it out into 2 different types of runs. One
which fixes the module versions so that we can tell if its the runtime that
causes regressions. A second runs with the latest versions of the modules
but a fixed runtime level so that we can identify regressions in the
modules themselves.

—
Reply to this email directly or view it on GitHub
#82 (comment).

[cjihrig]

One thought we had along the way would be if there was guidance from the Node community to module developers with respect to how to include tests so that they could be easily run.

I'm not sure if this is what you mean, but (as previously mentioned in this thread) npm install && npm test should be the only steps required for most modules.

[mhdawson]

@cjihrig, I've not looked into the details myself but from what I understand not all modules include the right scripts so that just works.

@CGavrila can you provide some of the specifics, possibly with an example of one where it does not work.