Mixing absolute and relative paths breaks native and custom formatters #1783
Description
Hi team!
Consider:
- a Node.js project using Cucumber.js at
/home/jan/project - a
featuresdirectory with anexample.featurefile inside it - my current working directory is
/home/jan/project - a Linux or mac OS machine (I haven't checked on Windows)
When I run the latest Cucumber 7.3.1 as follows:
cucumber-js features/example.feature /home/jan/project/features/example.feature
the following issues occur:
Issue 1 - feature files get executed twice
Cucumber doesn't understand that the relative features/example.feature is the same as the absolute /home/jan/project/features/example.feature and runs the feature twice.
Issue 2 - scenario location undefined
Location of the duplicate scenarios returned by parseTestCaseAttempt and formatterHelpers.PickleParser.getPickleLocation(testCaseAttempt) is undefined.
This breaks native formatters, such as snippets:
TypeError: Cannot read property 'line' of undefined
at Object.parseTestCaseAttempt (/home/jan/project/node_modules/@cucumber/cucumber/lib/formatter/helpers/test_case_attempt_parser.js:91:88)
at Object.formatTestCaseAttempt (/home/jan/project/node_modules/@cucumber/cucumber/lib/formatter/helpers/test_case_attempt_formatter.js:77:47)
at Object.formatIssue (/home/jan/project/node_modules/@cucumber/cucumber/lib/formatter/helpers/issue_helpers.js:26:68)
at /home/jan/project/node_modules/@cucumber/cucumber/lib/formatter/summary_formatter.js:56:32
at Array.forEach (<anonymous>)
at ProgressFormatter.logIssues (/home/jan/project/node_modules/@cucumber/cucumber/lib/formatter/summary_formatter.js:55:16)
at ProgressFormatter.logSummary (/home/jan/project/node_modules/@cucumber/cucumber/lib/formatter/summary_formatter.js:42:18)
at EventEmitter.<anonymous> (/home/jan/project/node_modules/@cucumber/cucumber/lib/formatter/summary_formatter.js:22:22)
at EventEmitter.emit (events.js:327:22)
at Runtime.start (/home/jan/project/node_modules/@cucumber/cucumber/lib/runtime/index.js:118:31)
at async Cli.run (/home/jan/project/node_modules/@cucumber/cucumber/lib/cli/index.js:205:23)
at async Object.run [as default] (/home/jan/project/node_modules/@cucumber/cucumber/lib/cli/run.js:25:18)
as well as custom formatters, such as Serenity/JS - serenity-js/serenity-js#975
Issue 3 - Incorrect messages emitted
testCaseStarted and related messages are emitted twice:
{"meta":{"protocolVersion":"16.0.1","implementation":{"name":"cucumber-js","version":"7.3.1"},"cpu":{"name":"x64"},"os":{"name":"darwin","version":"20.6.0"},"runtime":{"name":"node.js","version":"14.16.0"}}}
{"source":{"data":"Feature: An example feature\n\n Scenario: First\n\n Given a step that passes\n","uri":"features/example.feature","mediaType":"text/x.cucumber.gherkin+plain"}}
{"gherkinDocument":{"feature":{"tags":[],"location":{"line":1,"column":1},"language":"en","keyword":"Feature","name":"An example feature","description":"","children":[{"scenario":{"id":"9bce6771-fdcf-46ff-9794-c4fc89352f23","tags":[],"location":{"line":3,"column":3},"keyword":"Scenario","name":"First","description":"","steps":[{"id":"aacb2b99-c0e1-4e06-a24f-1334a981769c","location":{"line":5,"column":5},"keyword":"Given ","text":"a step that passes"}],"examples":[]}}]},"comments":[],"uri":"features/example.feature"}}
{"pickle":{"id":"6813e6ed-ddbe-4c01-897e-2a0c95fb1a14","uri":"features/example.feature","astNodeIds":["9bce6771-fdcf-46ff-9794-c4fc89352f23"],"tags":[],"name":"First","language":"en","steps":[{"id":"77e6d2a4-e5b7-49a9-89a2-645d1bf780c0","text":"a step that passes","astNodeIds":["aacb2b99-c0e1-4e06-a24f-1334a981769c"]}]}}
{"source":{"data":"Feature: An example feature\n\n Scenario: First\n\n Given a step that passes\n","uri":"features/example.feature","mediaType":"text/x.cucumber.gherkin+plain"}}
{"gherkinDocument":{"feature":{"tags":[],"location":{"line":1,"column":1},"language":"en","keyword":"Feature","name":"An example feature","description":"","children":[{"scenario":{"id":"c04a6aaf-ff3f-4606-938a-7e7661f40a1f","tags":[],"location":{"line":3,"column":3},"keyword":"Scenario","name":"First","description":"","steps":[{"id":"1663e51f-847a-4105-988e-1b25aa4097e8","location":{"line":5,"column":5},"keyword":"Given ","text":"a step that passes"}],"examples":[]}}]},"comments":[],"uri":"features/example.feature"}}
{"pickle":{"id":"1d34b7d3-935c-4c86-99fb-8a5e9abb60cb","uri":"features/example.feature","astNodeIds":["c04a6aaf-ff3f-4606-938a-7e7661f40a1f"],"tags":[],"name":"First","language":"en","steps":[{"id":"56e6eed4-3786-46fa-bd27-ffb77a763d27","text":"a step that passes","astNodeIds":["1663e51f-847a-4105-988e-1b25aa4097e8"]}]}}
{"testRunStarted":{"timestamp":{"seconds":1630374491,"nanos":498000000}}}
{"testCase":{"pickleId":"6813e6ed-ddbe-4c01-897e-2a0c95fb1a14","id":"fad8f086-04ab-45e1-a8f0-92439c05b79d","testSteps":[{"id":"d367e991-c27d-4975-9d00-f0992699f8d5","pickleStepId":"77e6d2a4-e5b7-49a9-89a2-645d1bf780c0","stepDefinitionIds":[],"stepMatchArgumentsLists":[]}]}}
{"testCase":{"pickleId":"1d34b7d3-935c-4c86-99fb-8a5e9abb60cb","id":"2e4befdd-9e94-4769-b8b7-c7ba1cf7b3e1","testSteps":[{"id":"e4a01c2b-d27e-4a98-b654-1ab6817c829d","pickleStepId":"56e6eed4-3786-46fa-bd27-ffb77a763d27","stepDefinitionIds":[],"stepMatchArgumentsLists":[]}]}}
{"testCaseStarted":{"attempt":0,"testCaseId":"fad8f086-04ab-45e1-a8f0-92439c05b79d","id":"dfa85e2f-cfb1-40ee-951b-37f22799c9da","timestamp":{"seconds":1630374491,"nanos":500000000}}}
{"testStepStarted":{"testCaseStartedId":"dfa85e2f-cfb1-40ee-951b-37f22799c9da","testStepId":"d367e991-c27d-4975-9d00-f0992699f8d5","timestamp":{"seconds":1630374491,"nanos":500000000}}}
{"testStepFinished":{"testCaseStartedId":"dfa85e2f-cfb1-40ee-951b-37f22799c9da","testStepId":"d367e991-c27d-4975-9d00-f0992699f8d5","testStepResult":{"status":"UNDEFINED","duration":{"seconds":0,"nanos":0},"willBeRetried":false},"timestamp":{"seconds":1630374491,"nanos":500000000}}}
{"testCaseFinished":{"testCaseStartedId":"dfa85e2f-cfb1-40ee-951b-37f22799c9da","timestamp":{"seconds":1630374491,"nanos":500000000}}}
{"testCaseStarted":{"attempt":0,"testCaseId":"2e4befdd-9e94-4769-b8b7-c7ba1cf7b3e1","id":"4dad91f9-2022-4942-8ce1-dd48314982d9","timestamp":{"seconds":1630374491,"nanos":501000000}}}
{"testStepStarted":{"testCaseStartedId":"4dad91f9-2022-4942-8ce1-dd48314982d9","testStepId":"e4a01c2b-d27e-4a98-b654-1ab6817c829d","timestamp":{"seconds":1630374491,"nanos":501000000}}}
{"testStepFinished":{"testCaseStartedId":"4dad91f9-2022-4942-8ce1-dd48314982d9","testStepId":"e4a01c2b-d27e-4a98-b654-1ab6817c829d","testStepResult":{"status":"UNDEFINED","duration":{"seconds":0,"nanos":0},"willBeRetried":false},"timestamp":{"seconds":1630374491,"nanos":501000000}}}
{"testCaseFinished":{"testCaseStartedId":"4dad91f9-2022-4942-8ce1-dd48314982d9","timestamp":{"seconds":1630374491,"nanos":501000000}}}
{"testRunFinished":{"timestamp":{"seconds":1630374491,"nanos":501000000},"success":false}}
Impact
The above issues are particularly problematic when Cucumber rerun feature is used (where the @rerun.txt file contains relative paths to feature files), together with absolute paths to other feature files, for example:
cucumber-js @rerun /home/jan/project/features/example.feature
where @rerun.txt:
features/example.feature:3
Also, when external frameworks such as Serenity/JS (maybe Stryker too? @nicojs?) allow users to specify paths to feature files as globs and then resolve those to absolute paths.
To Reproduce
See https://github.com/jan-molak/cucumber-rerun-bug
Note that in package.json I'm using realpath to get the absolute path dynamically. This will probably not work on Windows, but you could replace that with the actual absolute path when debugging.