Skip to content

Add known failure testing capability #38

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Dec 20, 2017
Merged

Add known failure testing capability #38

merged 3 commits into from
Dec 20, 2017

Conversation

tom-tan
Copy link
Member

@tom-tan tom-tan commented Dec 20, 2017

This request is to solve #37 by adding known failure capability to cwltest.

To use this feature, just add should_fail: true to tests in which the execution should fail.
This field is optional. When you do not specify should_fail field, cwltest treats that should_fail is false.
For example:

- doc: Test for section 11 (1st example)
  job: 11-records/record-job1.yml
  tool: 11-records/record.cwl
  should_fail: true

When should_fail is false, cwltest behaves as same as before. That is, it passes a test when the execution succeeds and it fails a test when the execution fails.
When should_fail is true, cwltest fails a test when the execution succeeds.
Here is an example message for it (I added # comments for explanation):

Test failed: cwl-runner --outdir=/var/folders/gk/zyzswjbs6m59ywz69zhrtmlh0000gn/T/tmp7dsw3jn4 --quiet 04-output/tar.cwl 04-output/tar-job.yml
Test for section 4 # corresponding `doc` field in YAML file
Returned zero but it should be non-zero # error message

Note: we may need other fields for more precise checking or there may be a better field name than should_fail.

@mr-c
Copy link
Member

mr-c commented Dec 20, 2017

👍 Can we get a test of this functionality?

@tom-tan
Copy link
Member Author

tom-tan commented Dec 20, 2017

I checked this request works by using CWL/user_guide repository.

Please save the following as _include/cwl/test1.yml in user_guide repository and run cwltest --test test1.yml in _include/cwl directory.

- doc: Test for section 4. This test will fail.
  job: 04-output/tar-job.yml
  tool: 04-output/tar.cwl
  should_fail: true

I used another test as follows. Please save it as _include/cwl/test2.yml in user_guide and run cwltest --test test2.yml in _include/cwl.

- doc: Test for section 11 (1st example). This test will succeed.
  job: 11-records/record-job1.yml
  tool: 11-records/record.cwl
  should_fail: true

@mr-c
Copy link
Member

mr-c commented Dec 20, 2017

Thanks @tom-tan! We can add that to this repository as part of another PR.

@mr-c mr-c mentioned this pull request Dec 20, 2017
@mr-c mr-c merged commit 4693cb9 into common-workflow-language:master Dec 20, 2017
@tom-tan tom-tan deleted the known-failure-testing branch December 21, 2017 01:01
@tom-tan
Copy link
Member Author

tom-tan commented Dec 21, 2017

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants