Implement targeted "overrides" of requirements on specific tools

[tetron]

You can provide an "overrides" section in your document:

cwltool revsort.cwl revsort-ovr-job.json

{
  "input": {
    "class": "File",
    "location": "whale.txt"
  },
    "cwltool:overrides": {
        "revtool.cwl": [{
            "class": "DockerRequirement",
            "dockerPull": "ubuntu:14.04"
        }]
    }
}

You can also use --overrides to pull in just the overrides section of a document:

cwltool --overrides revsort-ovr-job.json revsort.cwl revsort-job.json

    "cwltool:overrides": {
        "revtool.cwl": [{
            "class": "DockerRequirement",
            "dockerPull": "ubuntu:14.04"
        }]
    }

Currently only works on leaf nodes (CommandLineTool) but intended to generalize to workflows / workflow steps.

[tetron]

So the current implementation works by overriding the behavior of @get_requirement@, but this only works for leaf nodes and not for overrides applied to workflows or workflow steps.

So either:

1. Apply overrides at load time (this is where the requirements list gets built, so all the changes related to @get_requirement@ can be reverted)
2. Compute requirements at runtime, then apply overrides (requires refactoring)

The benefit of the 2nd approach is that you could, in principal, run multiple instances of the same workflow in the same process, but with different overrides, without loading multiple copies of the workflow/tool files. But it isn't clear we need that right now, and the refactoring is kind of annoying and will make this feature take longer.

[mr-c]

@tetron Option two will be useful later -- we've had request for the equivalent of scattering over requirements, for example.

Does the current syntax & option one allow for overriding a specific (sub)step (not to be confused with a specific CommandLineTool) ?

[tetron]

Yea, I was afraid you'd say that. I agree that it's better to apply it at runtime, just more work. But probably better to do it right than do it quick.

I hope to incorporate overrides into a future spec revision. That's an interesting idea, it could allow an expression to support the scatter case.

[mr-c]

Well, I'm not saying no to option 1 -- as long as it is forwards compatible with option 2 :-)

[tetron]

Implementation of proposal common-workflow-language/common-workflow-language#210

[mr-c]

Can you add an example of an identifier for a workflow step?

[tetron]

Added some discussion. Also made this more complex (but hopefully more useful) by resolving workflow ids relative to the workflow, and everything else relative to the job/--overrides document.

[mr-c]

I think this is over-indented