Add known failure testing capability #38
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This request is to solve #37 by adding known failure capability to
cwltest
.To use this feature, just add
should_fail: true
to tests in which the execution should fail.This field is optional. When you do not specify
should_fail
field,cwltest
treats thatshould_fail
is false.For example:
When
should_fail
is false,cwltest
behaves as same as before. That is, it passes a test when the execution succeeds and it fails a test when the execution fails.When
should_fail
is true,cwltest
fails a test when the execution succeeds.Here is an example message for it (I added
#
comments for explanation):Note: we may need other fields for more precise checking or there may be a better field name than
should_fail
.