multiCapabilities doesn't invoke parallel instances of WebDriver

[vsravuri]

Bug report
Using Protractor = 6.0.0-beta, multiCapabilities execute individual captibilities in sequential order rather than running the capabilities in parallel.

• Node Version: v10.15.1
• Protractor Version: 6.0.0-beta
• Selenium Standalone: 3.141.59
• Browser(s): Chrome 72/ Firefox 64
• Chrome Driver: 2.46
• Operating System and Version Mac OS 10.14.2

- Your protractor configuration file
// conf.js

exports.config = {
    framework: 'jasmine',
    seleniumAddress: 'http://localhost:4444/wd/hub',
    multiCapabilities: [{
        browserName: 'chrome',
        specs: ['spec.js'],
    }, {
        browserName: 'chrome',
        specs: ['spec.js'],
    }]
};

- A relevant example test
// spec.js

describe('Protractor Demo App',  () => {
    it('should have a title', async () => {
        await browser.get('http://juliemr.github.io/protractor-demo/');
        await browser.driver.sleep(5000);
        expect(await browser.getTitle()).toEqual('Super Calculator');
    });
});

- Output from running the test

➜  standard git:(vr-async-await) ✗ sudo ./node_modules/protractor/bin/protractor config.js                 
.[13:10:24] I/testLogger - 
------------------------------------
[13:10:24] I/testLogger - [chrome #01] PID: 11283
[chrome #01] Specs: /Users/vr/Documents/new-async-await/kb-automation/standard/spec.js
[chrome #01] 
[chrome #01] [13:10:13] I/local - Starting selenium standalone server...
[chrome #01] [13:10:14] I/local - Selenium standalone server started at http://192.168.0.11:56758/wd/hub
[chrome #01] DEPRECATION: Setting specFilter directly on Env is deprecated, please use the specFilter option in `configure`
[chrome #01] Randomized with seed 98392
[chrome #01] Started
[chrome #01] .
[chrome #01] 
[chrome #01] 
[chrome #01] 1 spec, 0 failures
[chrome #01] Finished in 6.914 seconds
[chrome #01] Randomized with seed 98392 (jasmine --random=true --seed=98392)

[13:10:24] I/testLogger - 

.[13:10:34] I/testLogger - 
------------------------------------

[13:10:34] I/testLogger - [chrome #11] PID: 11309
[chrome #11] Specs: /Users/vr/Documents/new-async-await/kb-automation/standard/spec.js
[chrome #11] 
[chrome #11] [13:10:24] I/local - Starting selenium standalone server...
[chrome #11] [13:10:25] I/local - Selenium standalone server started at http://192.168.0.11:62938/wd/hub
[chrome #11] DEPRECATION: Setting specFilter directly on Env is deprecated, please use the specFilter option in `configure`
[chrome #11] Randomized with seed 38299
[chrome #11] Started
[chrome #11] .
[chrome #11] 
[chrome #11] 
[chrome #11] 1 spec, 0 failures
[chrome #11] Finished in 6.914 seconds
[chrome #11] Randomized with seed 38299 (jasmine --random=true --seed=38299)

[13:10:34] I/testLogger - 

[13:10:34] I/launcher - 0 instance(s) of WebDriver still running
[13:10:34] I/launcher - 0 instance(s) of WebDriver still running
[13:10:34] I/launcher - Running 0 instances of WebDriver
[13:10:34] I/launcher - chrome #01 passed
[13:10:34] I/launcher - chrome #11 passed

Able to reproduce this issue only in Protector 6.0.0-beta

[vsravuri]

@cnishina

Could you please have a look at this issue? This issue is blocking me from running automation tests using Protractor = 6.0.0-beta branch.

[vsravuri]

@cnishina @heathkit @juliemr

Could you please have a look at this issue? I am unable to run my tests with multiCapabilities using 6.0.0-beta build

[vsravuri]

@cnishina

Any reason why you think multiCapabilities not working in Protractor 6.0.0?

[pkearney13]

Same here

[jithinkmatthew]

Facing same problem with protractor version 6.0.0

[vsravuri]

@cnishina

Any update on this issue?

[cnishina]

Yikes, this appears to be an issue. Maybe keep using 5.4.2 until this is resolved. I'm not sure and will talk it over with Julie.

[cnishina]

While removing q from Protractor, I did run into this (which might be suspect): https://github.com/angular/protractor/blob/master/lib/launcher.ts#L231

My problem with this was we called a method that returned a Promise without awaiting the promise. I ended up putting an await there. This might not be it... It might be:

https://github.com/angular/protractor/blob/master/lib/launcher.ts#L250

[vsravuri]

@cnishina

Appreciate your help for looking into this issue. I will continue using 5.4.2 until this issue is resolved.

[vsravuri]

@juliemr @cnishina

Any plan to fix this issue in the beta release you are working on?

[vsravuri]

@cnishina @juliemr

Both multiCapabilities & shardTestFiles (#5232) doesn't work with Protractor 6.0.0

Eagerly waiting for next release of Protractor to unblock in running parallel tests.

[cnishina]

Similar issue here: #5232

I have not had time to look at this.

[gawicks]

Any updates on this?

[telangn]

I think this is the same issue, my error is saying the path to the spec file for chrome is not correct, but I believe it is looking at my project structure, it's obvious. So I believe the error is wrong, but it is the same issue.

Conf file:

In my multiCapabilities I have two different browsers with the argument

**specs: ./e2e/file-presentation-flow/*.e2e-spec.ts**

and

**specs: './e2e/smoke-tests/*.e2e-spec.ts**

respectively, but only Firefox browser launches. I expect both Chrome and Firefox to run parallel (2 instances of webDriver) with different spec files respectively.

    browserName: 'chrome',
    chromeOptions: {
      args: ['--no-sandbox', '--disable-gpu', '--disable-dev-shm-usage', '--use-fake-ui-for-media-stream',
        '--ignore-certificate-errors', '--allow-insecure-localhost', '--incognito', '--headless', '--start-maximized', '--user-agent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36"'],
      prefs: {
        download: {
          prompt_for_download: false,
          default_directory: process.env.DOWNLOAD_DIRECTORY
        },
        specs: './e2e/file-presentation-flow/*.e2e-spec.ts',
        'profile.content_settings.exceptions.clipboard': {
          'https://files.globalmeet.com,*': { last_modified: Date.now(), setting: 1 }
        },
      }
    },
  },
    {
      browserName: 'firefox',
      'moz:firefoxOptions': {
        'args': ['--safe-mode', '--headless']
      },
      specs: './e2e/smoke-tests/*.e2e-spec.ts'
    }
  ],```

**LOGS:**

```[10:30:35] I/testLogger - [chrome #01] PID: 21345
[chrome #01] /home/ntelang/PGi/convergence-client/node_modules/protractor/built/runner.js:322
[chrome #01]             throw new Error('Spec patterns did not match any files.');
[chrome #01]             ^
[chrome #01] 
[chrome #01] Error: Spec patterns did not match any files.
[chrome #01]     at Runner.run (/home/ntelang/PGi/convergence-client/node_modules/protractor/built/runner.js:322:19)
[chrome #01]     at process.on (/home/ntelang/PGi/convergence-client/node_modules/protractor/built/runnerCli.js:43:20)
[chrome #01]     at emitTwo (events.js:126:13)
[chrome #01]     at process.emit (events.js:214:7)
[chrome #01]     at emit (internal/child_process.js:772:12)
[chrome #01]     at _combinedTickCallback (internal/process/next_tick.js:141:11)
[chrome #01]     at process._tickCallback (internal/process/next_tick.js:180:9)
[chrome #01] 


[10:30:35] I/testLogger - 

[10:30:35] E/launcher - Runner process exited unexpectedly with error code: 1
[10:30:35] I/launcher - 1 instance(s) of WebDriver still running
..[10:31:08] I/testLogger - 
------------------------------------

[10:31:08] I/testLogger - [firefox #11] PID: 21350
[firefox #11] Specs: /home/ntelang/PGi/convergence-client/e2e/smoke-tests/smokeTest.e2e-spec.ts
[firefox #11] 
[firefox #11] [10:30:35] I/direct - Using FirefoxDriver directly...
[firefox #11] Spec started

[wodddy]

When should we expect 6.0.1? It looks like the development of a protractor has stopped completely. And the latest version 6.0.0 has a typescript error. that makes it totally unuseful. Can somebody release just that fix under 6.0.1, and then take care of the rest in a scope of 6.0.2?

Or else, if there's no more development of protractor, can it be announced, so that people start to look for alternatives?

[froblesmartin]

Any update on this? Has someone found any workaround in the meantime? @cnishina

Even more important: Is the project abandoned?

[abruslik]

I've found workaround for parallel sessions using bluebird promises.
Bluebird allows to control count of concurrent promises.
But you should know that it was not tested across all available multiCapabilities configurations.
In my case it just fixed configuration with shardTestFiles and maxInstances attributes.
launcher.js:

"use strict";
var __awaiter = (this && this.__awaiter) || function (thisArg, _arguments, P, generator) {
	return new (P || (P = Promise))(function (resolve, reject) {
		function fulfilled(value) {
			try {
				step(generator.next(value));
			} catch (e) {
				reject(e);
			}
		}

		function rejected(value) {
			try {
				step(generator["throw"](value));
			} catch (e) {
				reject(e);
			}
		}

		function step(result) {
			result.done ? resolve(result.value) : new P(function (resolve) {
				resolve(result.value);
			}).then(fulfilled, rejected);
		}

		step((generator = generator.apply(thisArg, _arguments || [])).next());
	});
};
Object.defineProperty(exports, "__esModule", {value: true});
/**
 * The launcher is responsible for parsing the capabilities from the
 * input configuration and launching test runners.
 */
const fs = require("fs");
const configParser_1 = require("./configParser");
const exitCodes_1 = require("./exitCodes");
const logger_1 = require("./logger");
const taskRunner_1 = require("./taskRunner");
const taskScheduler_1 = require("./taskScheduler");
const util_1 = require("./util");
const promiseBluebird = require('bluebird');
let logger = new logger_1.Logger('launcher');
let RUNNERS_FAILED_EXIT_CODE = 100;

/**
 * Keeps track of a list of task results. Provides method to add a new
 * result, aggregate the results into a summary, count failures,
 * and save results into a JSON file.
 */
class TaskResults {
	constructor() {
		// TODO: set a type for result
		this.results_ = [];
	}

	add(result) {
		this.results_.push(result);
	}

	totalSpecFailures() {
		return this.results_.reduce((specFailures, result) => {
			return specFailures + result.failedCount;
		}, 0);
	}

	totalProcessFailures() {
		return this.results_.reduce((processFailures, result) => {
			return !result.failedCount && result.exitCode !== 0 ? processFailures + 1 : processFailures;
		}, 0);
	}

	saveResults(filepath) {
		let jsonOutput = this.results_.reduce((jsonOutput, result) => {
			return jsonOutput.concat(result.specResults);
		}, []);
		let json = JSON.stringify(jsonOutput, null, '  ');
		fs.writeFileSync(filepath, json);
	}

	reportSummary() {
		let specFailures = this.totalSpecFailures();
		let processFailures = this.totalProcessFailures();
		this.results_.forEach((result) => {
			let capabilities = result.capabilities;
			let shortName = (capabilities.browserName) ? capabilities.browserName : '';
			shortName = (capabilities.logName) ?
				capabilities.logName :
				(capabilities.browserName) ? capabilities.browserName : '';
			shortName += (capabilities.version) ? capabilities.version : '';
			shortName += (capabilities.logName && capabilities.count < 2) ? '' : ' #' + result.taskId;
			if (result.failedCount) {
				logger.info(shortName + ' failed ' + result.failedCount + ' test(s)');
			} else if (result.exitCode !== 0) {
				logger.info(shortName + ' failed with exit code: ' + result.exitCode);
			} else {
				logger.info(shortName + ' passed');
			}
		});
		if (specFailures && processFailures) {
			logger.info('overall: ' + specFailures + ' failed spec(s) and ' + processFailures +
				' process(es) failed to complete');
		} else if (specFailures) {
			logger.info('overall: ' + specFailures + ' failed spec(s)');
		} else if (processFailures) {
			logger.info('overall: ' + processFailures + ' process(es) failed to complete');
		}
	}
}

let allTasksCount;
let taskResults_ = new TaskResults();
/**
 * Initialize and run the tests.
 * Exits with 1 on test failure, and RUNNERS_FAILED_EXIT_CODE on unexpected
 * failures.
 *
 * @param {string=} configFile
 * @param {Object=} additionalConfig
 */
let initFn = function (configFile, additionalConfig) {
	return __awaiter(this, void 0, void 0, function* () {
		let configParser = new configParser_1.ConfigParser();
		if (configFile) {
			configParser.addFileConfig(configFile);
		}
		if (additionalConfig) {
			configParser.addConfig(additionalConfig);
		}
		let config = configParser.getConfig();
		logger_1.Logger.set(config);
		logger.debug('Running with --troubleshoot');
		logger.debug('Protractor version: ' + require('../package.json').version);
		logger.debug('Your base url for tests is ' + config.baseUrl);
		// Run beforeLaunch
		yield util_1.runFilenameOrFn_(config.configDir, config.beforeLaunch);
		// 1) If getMultiCapabilities is set, resolve that as
		// `multiCapabilities`.
		if (config.getMultiCapabilities && typeof config.getMultiCapabilities === 'function') {
			if (config.multiCapabilities.length || config.capabilities) {
				logger.warn('getMultiCapabilities() will override both capabilities ' +
					'and multiCapabilities');
			}
			// If getMultiCapabilities is defined and a function, use this.
			const waitMultiConfig = yield config.getMultiCapabilities();
			config.multiCapabilities = waitMultiConfig;
			config.capabilities = null;
		}
		// 2) Set `multicapabilities` using `capabilities`,
		// `multicapabilities`, or default
		if (config.capabilities) {
			if (config.multiCapabilities.length) {
				logger.warn('You have specified both capabilities and ' +
					'multiCapabilities. This will result in capabilities being ' +
					'ignored');
			} else {
				// Use capabilities if multiCapabilities is empty.
				config.multiCapabilities = [config.capabilities];
			}
		} else if (!config.multiCapabilities.length) {
			// Default to chrome if no capabilities given
			config.multiCapabilities = [{browserName: 'chrome'}];
		}
		// 3) If we're in `elementExplorer` mode, throw an error and exit.
		if (config.elementExplorer || config.framework === 'explorer') {
			const err = new Error('Deprecated: Element explorer depends on the ' +
				'WebDriver control flow, and thus is no longer supported.');
			logger.error(err);
			process.exit(1);
		}
		// 4) Run tests.
		let scheduler = new taskScheduler_1.TaskScheduler(config);
		if (!allTasksCount){
			allTasksCount = scheduler.numTasksOutstanding();
		}
		process.on('uncaughtException', (exc) => {
			let e = (exc instanceof Error) ? exc : new Error(exc);
			if (config.ignoreUncaughtExceptions) {
				// This can be a sign of a bug in the test framework, that it may
				// not be handling WebDriver errors properly. However, we don't
				// want these errors to prevent running the tests.
				logger.warn('Ignoring uncaught error ' + exc);
				return;
			}
			logger.error(e.message);
			logger.error(e.stack);
			if (e instanceof exitCodes_1.ProtractorError) {
				let protractorError = e;
				process.exit(protractorError.code);
			} else {
				process.exit(1);
			}
		});
		process.on('unhandledRejection', (reason, p) => {
			if (reason.stack.match('angular testability are undefined') ||
				reason.stack.match('angular is not defined')) {
				logger.warn('Unhandled promise rejection error: This is usually occurs ' +
					'when a browser.get call is made and a previous async call was ' +
					'not awaited');
			}
			logger.warn(p);
		});
		process.on('exit', (code) => {
			if (code) {
				logger.error('Process exited with error code ' + code);
			} else if (scheduler.numTasksOutstanding() > 0) {
				logger.error('BUG: launcher exited with ' + scheduler.numTasksOutstanding() + ' tasks remaining');
				process.exit(RUNNERS_FAILED_EXIT_CODE);
			}
		});
		// Run afterlaunch and exit
		const cleanUpAndExit = (exitCode) => __awaiter(this, void 0, void 0, function* () {
			try {
				const returned = yield util_1.runFilenameOrFn_(config.configDir, config.afterLaunch, [exitCode]);
				if (typeof returned === 'number') {
					process.exit(returned);
				} else {
					process.exit(exitCode);
				}
			} catch (err) {
				logger.error('Error:', err);
				process.exit(1);
			}
		});
		const totalTasks = scheduler.numTasksOutstanding();
		let forkProcess = false;
		if (totalTasks > 1) { // Start new processes only if there are >1 tasks.
			forkProcess = true;
			if (config.debug) {
				throw new exitCodes_1.ConfigError(logger, 'Cannot run in debug mode with multiCapabilities, count > 1, or sharding');
			}
		}
		const createNextTaskRunner = () => __awaiter(this, void 0, void 0, function* () {
			return new Promise((resolve) => __awaiter(this, void 0, void 0, function* () {
				const task = scheduler.nextTask();
				if (task) {
					const taskRunner = new taskRunner_1.TaskRunner(configFile, additionalConfig, task, forkProcess);
					try {
						const result = yield taskRunner.run();
						if (result.exitCode && !result.failedCount) {
							logger.error('Runner process exited unexpectedly with error code: ' + result.exitCode);
						}
						taskResults_.add(result);
						task.done();
						// yield createNextTaskRunner();
						// If all tasks are finished
						// if (scheduler.numTasksOutstanding() === 0) {
							resolve();
						// }
						logger.info(scheduler.countActiveTasks() + ' instance(s) of WebDriver still running');
					} catch (err) {
						const errorCode = exitCodes_1.ErrorHandler.parseError(err);
						logger.error('Error:', err.stack || err.message || err);
						yield cleanUpAndExit(errorCode ? errorCode : RUNNERS_FAILED_EXIT_CODE);
					}
				} else {
					resolve();
				}
			}));
		});
		const taskList = Array(allTasksCount).fill(() => createNextTaskRunner());
		const maxConcurrentTasks = scheduler.maxConcurrentTasks();
		yield promiseBluebird.map(taskList, task => task(), {concurrency: maxConcurrentTasks});
		// for (let i = 0; i < maxConcurrentTasks; ++i) {
		//     yield createNextTaskRunner();
		// }
		logger.info('Running ' + scheduler.countActiveTasks() + ' instances of WebDriver');
		// By now all runners have completed.
		// Save results if desired
		if (config.resultJsonOutputFile) {
			taskResults_.saveResults(config.resultJsonOutputFile);
		}
		taskResults_.reportSummary();
		let exitCode = 0;
		if (taskResults_.totalProcessFailures() > 0) {
			exitCode = RUNNERS_FAILED_EXIT_CODE;
		} else if (taskResults_.totalSpecFailures() > 0) {
			exitCode = 1;
		}
		yield cleanUpAndExit(exitCode);
		// Start `const maxConcurrentTasks` workers for handling tasks in
		// the beginning. As a worker finishes a task, it will pick up the next
		// task from the scheduler's queue until all tasks are gone.
	});
};
exports.init = initFn;
//# sourceMappingURL=launcher.js.map

[Phonesis]

@cnishina Keen to know where things stand with Protractor and future of version 6. Not seen any updates for few months?

[froblesmartin]

I was trying to contact some developers of this project and Angular 4 months ago and I had no reply. It really seems abandoned.

[Kungfoowiz]

Yes, this is important. Please announce if protractor development is being continued or we need to look into alternatives?

[telangn]

Start using Cypress

[Phonesis]

I don't see Cypress as a viable alternative atm. It only supports Chrome and doesn't work with BrowserStack. Also, it requires a paid license.

[vsravuri]

@heathkit @cnishina @juliemr

Appreciate if you could provide some insight into the future of Protractor. Its' been more than 6 months since we have any update on Protractor 6.0. If the plan is to abandon protractor, kindly let us know so that we have enough room to look for the alternatives.

[yadimon]

While removing q from Protractor, I did run into this (which might be suspect): https://github.com/angular/protractor/blob/master/lib/launcher.ts#L231

My problem with this was we called a method that returned a Promise without awaiting the promise. I ended up putting an await there. This might not be it... It might be:

https://github.com/angular/protractor/blob/master/lib/launcher.ts#L250

here you wait for every runner, sequentially in the loop.
now:

for (let i = 0; i < maxConcurrentTasks; ++i) {
  await createNextTaskRunner();
}

should be:

const runners = [];
for (let i = 0; i < maxConcurrentTasks; ++i) {
  runners.push(createNextTaskRunner());
}
await Promise.all(runners); // wait for all, running simultaneously

and Promise((resolve) => ...) (https://github.com/angular/protractor/blob/master/lib/launcher.ts#L219)
should be just async function, so remove the line. Otherwise, no resolve() happens at the end of the method with outstanding tasks https://github.com/angular/protractor/blob/master/lib/launcher.ts#L235

[heathkit]

@heathkit @cnishina @juliemr no longer work on Protractor, we can't speak to future plans. We continue to use Protractor internally, and there's a lot of other projects at Google that depend on it.

The Angular team would like to invest more in Protractor, and will have time now that they've landed Ivy. @IgorMinar will be better able to answer questions about the future of Protractor.

[rajjaiswalsaumya]

Is someone fixing this ? I need it badly. Our tests serially takes more than an hour. Please help.