Option strict not parsed: cucumber.execution.strict

[rubienr]

🤔 What's the problem you're trying to solve?

GIVEN I use Cucumber with Scala and SBT
AND I have dependency to cucumber-scala 8.13.1
AND SBT plugin com.waioeka.sbt.cucumber-plugin 0.3.1 is enabled
WHEN I want tests not to fail if steps are not implemented
THEN I am not able to successfully set the Cucumber option strict to false

✨ What's your proposed solution?

I would expect the CucumberPropertiesParser.java supports parsing this option so that I at least can pass this option by defining it in the cucumber.properties file.

⛏ Have you considered any alternatives or workarounds?

I tried to pass the option as follows:

• cucumber.properties file: cucumber.execution.strict=false
• setting environment variable from the SBT plugin cofiguration: CucumberPlugin.envProperties := Map("CUCUMBER_EXECUTION_STRICT"->"false")
• passing cli argument from SBT command: sbt cucmber -- --strict=false)

[mpkorstanje]

Non strict execution caused a lot of friction with other tooling (for example see cucumber-junit-xml-formatter#test-outcome-mapping) so we are in the process of deprecating and removing it from all Cucumber implementations. Cucumber JVM has already gone through this cycle. You can read more about the rationale in #1788 and cucumber/common#714.

[rubienr]

Is it possible to somehow mimic this behaviour without --strict (and without tag filtering)? I also don't understand the difference in-between 'undefined' and 'pending' in the table at cucumber/common/issues/714.

The procedure I tried is:

1. write a step WHEN this step is not implemented, leave it undefined
2. run Cucumber in sbt: sbt:root> cucumber
3. copy paste the snippet from the test result message:

   When("""this step is not implemented""") { () =>
   // Write code here that turns the phrase above into concrete actions
   throw new io.cucumber.scala.PendingException()
   }

4. re-run Cucumber in sbt
5. i still see it failing

   [error] stack trace is suppressed; run last <module-name> / cucumber for the full output
   [error] (<module-name> / cucumber) java.lang.IllegalStateException: Cucumber did not succeed and returned error =1
   [error] Total time: 1 s, completed Jan 12, 2023, 10:13:03 AM

6. and the respective step in the HTML-test report is red

I would want

• the undefined step to be yellow in the report (not red) and
• to be able to distinguish in sbt (for example return code from cucumber)
  • a) if the test run failed
  • b) or all were passed, except some were not implemented.

[mpkorstanje]

Is it possible to somehow mimic this behaviour without --strict (and without tag filtering)?

No. The CLI will exit with a non-zero code for any scenarios that have steps that aren't passed or skipped. If we don't follow this pattern we make it more difficult for the ecosystem that process reports.

the undefined step to be yellow in the report (not red)

I think that would be sensible. I would expect each status to have a somewhat distinct color. They're defined in:

https://github.com/cucumber/react-components/blob/main/src/components/app/StatusesSummary.module.scss

Perhaps you can send a pull request, my react skills are practicality non-existent.

[rubienr]

No. The CLI will exit with a non-zero code for any scenarios that have steps that aren't passed or skipped. If we don't follow this pattern we make it more difficult for the ecosystem that process reports.

I see.

I think that would be sensible. I would expect each status to have a somewhat distinct color. They're defined in:

https://github.com/cucumber/react-components/blob/main/src/components/app/StatusesSummary.module.scss

Perhaps you can send a pull request, my react skills are practicality non-existent.

Well the predefined colors seem fine. I would probably change the leading symbol from red-X to organge-X. Unfortunately I have absolutely no react skills.

I think we can close this issue, since the most important part for me would have been to not fail if steps are not implemented. I understand the decision and the reasoning to deprecate 'strict'.

Thank you very much for the fast replies and quick clarification.