Support Sharding

[juliemr]

This has come up a couple times, so I wanted to consolidate discussion into one issue.

Test sharding would mean automagically splitting up a suite of tests to run in parallel, with the goal of reducing test time. Implementation will be very dependent on the test runner, and so would probably happen for Jasmine first.

Currently, you can achieve this by splitting tests into multiple files and creating different configs for different specs.

[elgalu]

👍

[awal112358]

Hey Julie,

How granular were you thinking this would need to be initially? If it doesn't need to be super-configurable, I could imagine something like:

maxSessions: < int >

• this would drive how many simultaneous runs could occur.
• defaults to 0 - meaning no limitation.

splitSpecsBetweenCapabilities: < bool >

• false (default)
  • runs all specs matched for EVERY capability specified.
• true
  • runs all specs matched ONCE, shared across capabilities.

Description:

• We would change the launcher/runner system so that we keep track of which capabilities are currently running and which are available and we only launch a new capability run when maxSessions is > current running capabilities. Through messaging the launcher would know when capabilities are in progress or idle and available and would make use of them in a FIFO manner.
• It might be cleaner to make specOrder and capabilityMode properties of multiCapabilities. For example:

multiCapabilities: {
  split: false,
  capabilities: [
    browserName: 'firefox',
    count: 2
  ]
}

Use Cases

---

all scenarios assume 6 tests total.
all scenarios assume each test takes 10 seconds.

Story 1:

"I want to run all of my tests in order across firefox and chrome at the same time."

• Solution:
  • split: false
  • maxSessions: 2+ (or falsey for unlimited)
• Result:
  • 6 tests will run sequentially for firefox and chrome at the same time.
  • 12 total tests (120 seconds)
  • 60 seconds duration. (speedup 2x)

Story 2:

"I want to run my tests in order across firefox and chrome, but throttled because my hosted solution or computer, etc limits me to testing 1 at a time."

• Solution:
  • split: false
  • maxSessions: 1
• Result:
  • 6 tests will run sequentially for firefox. When these are done, 6 tests will run in order on chrome.
  • 12 total tests (120 seconds)
  • 120 seconds duration. (no speedup)

Story 3:

"I want to run all my tests as fast as possible and they don't need to be in order. I have 3 firefox and 3 chrome instances available and don't care how they are distributed as long as they finish ASAP."

• Solution:
  • split: true
  • maxSessions: 6 (or falsey for unlimited)
• Result:
  • 6 tests will run between 3 firefox and 3 chrome instances. Although they will come on a first-available, first-served order, odds are this would mean 1 test for each capability.
  • 6 total tests (60 seconds)
  • 10 seconds duration. (6x speedup)

[juliemr]

Thanks for the writeup!

I don't think that we should support story 4. That would mean that 'capabilityMode' and 'specOrder' could be condensed into one option, which means either 'run each test once on each configuration' or 'run each test once total, split up the tests between configurations'. Let's call it 'shardTestsBetweenConfigurations' or something? I'm trying to think of the least confusing name :)

[awal112358]

Sure :)

I updated it to be simpler and picked a term I hope is easy to understand, but let me know if you have something else.

Also - I'm torn between whether it should be part of the multiCapabilities.

• The argument for it being part of the multiCapabilities config option is that it helps to prevent invalid config combinations (such as splitSpecsBetweenCapabilities: true + just having capabilities - not multiCapabilities.
• The argument against it would be that we would have to change the format of multiCapabilities to be an object with an array property inside of it, which either creates a breaking change OR forces us to add code to support a polymorphic value here :(

Any thoughts?

[juliemr]

I like split! I like the verbose splitTestsBeweenCapabilities version of the name - I'd like to err on the side of making sure the user knows exactly what they're doing.

As for multiCapabilities, I would say make it separate. I think having multiCapabilities as an array makes more sense, and I'm not particularly concerned about the invalid combination - to me, splitSpecsBetweenCapabilities: true and just one capabilities means split the tests between the one available capability, problem solved :)

[awal112358]

Awesome! I'll see if I can knock this out relatively soon. Thanks :)

[juliemr]

Cool - and just a heads up in case things change unexpectedly under you, I'm planning on working on #542 later this week, which will involve changes to the runner and launcher so that they send messages based on tests being completed.

[awal112358]

Thanks for the heads up - that implementation will make this a lot easier. I'll hold off for a bit. I'm excited to see it :)

[FrigoEU]

I tried my hand at implementing this as awal112358 described above. PR: #655

[bytasv]

Hey, what is the status of test splitting between capabilities? Is anyone working on it, maybe I could help with something because I would love to see this feature landing soon

[hankduan]

@rafaelbattesti adding something like

  multiCapabilities: [{
    'browserName': 'chrome'
  }, {
    'browserName': 'firefox'
  }],

will allow you to run a set of tests on multiple browsers. Then just refactor your code to sign in with two different users.
@bytasv #655 should achieve what you need