Run in worker threads

[Qix-]

See #1428 (comment)

---

Right now it appears the process is forked for each test and then the test is ran. This is unideal for a vast amount of reasons.

Upon discussion with Sindre, it was decided that we should explore moving into a pooled approach (where instances of the following classes shared a common interface and abstracted away the strategy for executing individual tests):

• SingleProcessTestPool would run all tests in the main process without forking
• ForkTestPool would fork the process N times (where N is the number of processors) and distribute tests out between them as they become idle

In the future, this could be expanded to a WorkerTestPool to implement support for workers.

This would give us the ability to decide upon the execution strategy based on a number of factors:

• For <N tests, use SingleProcessTestPool; otherwise, use ForkTestPool
• When --no-fork or equivalent flag is provided, force SingleProcessTestPool
• When --inspect or --inspect-brk is specified, force SingleProcessTestPool (ref Debugging with the Node.js debugger #1108)

I'm not entirely sure how much work this would require, but consider this issue a 'jumping off' point for discussion.

EDIT: This might be of interest as well: https://nodejs.org/api/worker_threads.html

---

IssueHunt Summary

dnlup has been rewarded.

Backers (Total: $300.00)

• sindresorhus ($60.00)
• issuehunt ($140.00)
• futpib ($100.00)

Submitted pull Requests

• #2690 feat: run in worker threads

---

Tips

• Checkout the Issuehunt explorer to discover more funded issues.
• Need some help from other developers? Add your repositories on IssueHunt to raise funds.

[sindresorhus]

• SingleProcessTestPool would run all tests in the main process without forking

Would solve #1332.

[unional]

FYI on Jest using workers and also spread work based on the previous test duration:
http://facebook.github.io/jest/blog/2016/03/11/javascript-unit-testing-performance.html

[sindresorhus]

@unional Yes, that's a good way to handle it, and with the design @Qix- is describing, it shouldn't be too hard to do.

[IssueHuntBot]

@sindresorhus has funded $60.00 to this issue. See it on IssueHunt

[Qix-]

Issue hunt link is auth-walled :/ Any way to see it without associating it with my account?

[Qix-]

Edited the OP to mention https://nodejs.org/api/worker_threads.html.

[IssueHuntBot]

@issuehuntfest has funded $200.00 to this issue. See it on IssueHunt

[issuehunt-oss]

@futpib has funded $100.00 to this issue.

---

• Submit pull request via IssueHunt to receive this reward.
• Want to contribute? Chip in to this issue via IssueHunt.
• Checkout the IssueHunt Issue Explorer to see more funded issues.
• Need help from developers? Add your repository on IssueHunt to raise funds.

[novemberborn]

We should only support worker threads and Node.js 12.16 or higher. I think it could even be the default on supported platforms. Despite the "pool" concepts discussed here, you'll have a better chance of getting this shipped by modifying lib/fork.js.

[sibelius]

what about using jest-worker for this?

[dnlup]

👋🏻 @novemberborn

I have worked on this issue, and I should have something ready to review by next week. I have not used a pooled approach, to keep things more similar to the current strategy.

[novemberborn]

@dnlup if you submit your PR on IssueHunt (I'm not entirely sure how) there's 270 USD waiting for you 😉

[dnlup]

@dnlup if you submit your PR on IssueHunt (I'm not entirely sure how) there's 270 USD waiting for you

😄 I did submit it, maybe I have to submit it again?

[issuehunt-oss]

@novemberborn has rewarded $270.00 to @dnlup. See it on IssueHunt

• 💰 Total deposit: $300.00
• 🎉 Repository reward(0%): $0.00
• 🔧 Service fee(10%): $30.00

[Qix-]

Incredible work here, thank you @dnlup.

---

I can't tell who, but <del> tags have magically appeared in the OP post here, and I'm not sure who did that. I don't use those tags (didn't even know they existed) so it certainly wasn't me.

Can we please not get in the habit of doing that? I didn't write all that just for it to be strike-through'd, and it's quite annoying to have someone else manhandle my text.

Just a sidenote.

[novemberborn]

I can't tell who, but <del> tags have magically appeared in the OP post here, and I'm not sure who did that.

I can't tell from the edit history either, very strange. I've removed them.

[dnlup]

Incredible work here, thank you @dnlup.

I can't tell who, but <del> tags have magically appeared in the OP post here, and I'm not sure who did that. I don't use those tags (didn't even know they existed) so it certainly wasn't me.

Can we please not get in the habit of doing that? I didn't write all that just for it to be strike-through'd, and it's quite annoying to have someone else manhandle my text.

Just a sidenote.

thank you @Qix-