Use existing location if a secondary file has one.

[DailyDreaming]

This allows Toil to pass v1.2 conformance test 307.

In Toil, we pass the following (example) inputs to the executor:

{
    "inputWithSecondary": {
        "class": "File",
        "location": "toilfs:0:files/no-job/file-1120f3b05d62444083a9ff5c60c7f772/secondary_file_test.txt",
        "size": 0,
        "basename": "secondary_file_test.txt",
        "nameroot": "secondary_file_test",
        "nameext": ".txt",
        "secondaryFiles": [
            {
                "location": "toilfs:0:files/no-job/file-ccead684521d44f98ff81b118f37159b/secondary_file_test.txt.accessory",
                "class": "File",
                "basename": "secondary_file_test.accessory",
                "nameroot": "secondary_file_test",
                "nameext": ".accessory",
                "size": 0
            }
        ]
    }
}

When cwltool makes its builder, it swaps:

toilfs:0:files/no-job/file-ccead684521d44f98ff81b118f37159b/secondary_file_test.txt.accessory

for

toilfs:0:files/no-job/file-1120f3b05d62444083a9ff5c60c7f772/secondary_file_test.txt.accessory

The bolded portion is the same basedir as the primary file, which creates a path to a file that does not exist.

It does this here:

cwltool/cwltool/builder.py

Line 428 in 78fe9d4

d_location[0 : d_location.rindex("/") + 1]

This seems to be because cwltool:

1. Assumes that primary and secondary files are in the same directory and thus have the same interchangeable basedir in the toilfs: schema.
2. Does not check to see if secondaryfiles have been resolved yet. (the list has resolved secondary files in toil because we process with a builder already before it runs a builder again in cwltool)

Two possible proposed solutions:

• The easier solution imo (this PR) is to check for an already processed secondary file before fabricating a new secondary file's entry.
• An alternative solution might be to force toil to place secondary files at the same basedir as the primary file if the first solution isn't preferable.

[codecov]

Codecov Report

Merging #1380 (f40a017) into main (78fe9d4) will decrease coverage by 0.07%.
The diff coverage is 7.14%.

@@            Coverage Diff             @@
##             main    #1380      +/-   ##
==========================================
- Coverage   54.25%   54.17%   -0.08%     
==========================================
  Files          42       42              
  Lines        7767     7780      +13     
  Branches     1976     1984       +8     
==========================================
+ Hits         4214     4215       +1     
- Misses       2970     2981      +11     
- Partials      583      584       +1     

Impacted Files	Coverage Δ
cwltool/builder.py	45.48% <7.14%> (-1.25%)	⬇️
cwltool/job.py	41.05% <0.00%> (-0.30%)	⬇️
cwltool/process.py	69.46% <0.00%> (+0.04%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 78fe9d4...f40a017. Read the comment docs.

[mr-c]

Can you run make mypy and address the issues?

https://travis-ci.com/github/common-workflow-language/cwltool/jobs/453580948#L628

[DailyDreaming]

@mr-c The linting should be addressed (although the checking is perhaps a bit verbose; please advise if I you'd prefer I handle it differently). One test is failing because run_test.sh in cwl-v1.2 no longer has executable permissions. I made a PR here and tests should pass once it's merged: common-workflow-language/cwl-v1.2#74

[mr-c]

We are out of Travis CI coins, sorry @DailyDreaming ; If you have Travis CI coins in your personal account, you could fork the repo and run them there

[DailyDreaming]

@mr-c Sorry I used your coins. Will be much more conservative about running tests in the future.

All tests passed when run earlier except for the cwl-v1.2 tests, which passed locally for me after the execution bit fix. I'll just wait on Travis here then.

[mr-c]

@DailyDreaming Don't worry about it! There are a lot of projects that pull from the same bucket of coins.

[DailyDreaming]

Seems like this may have been fixed already. Updating cwltool in Toil seems to have resolved this issue. Closing!