Docker: directory mount fails

[michael-kotliar]

Tested for cwltool 1.0.20170327143622
Docker is trying to mount not only File but Directory too. When Directory is returned from expression tool it fails to mount.
Testing example: https://github.com/michael-kotliar/cwl_sandbox/tree/master/docker_mount
To run: cwltool --debug workflow.cwl job.yml

Detailed log

  
   Resolved 'workflow.cwl' to 'file:///Users/kot4or/workspaces/cwl_ws/cwl_sandbox/docker_mount/workflow.cwl'
[workflow workflow.cwl] initialized from file:///Users/kot4or/workspaces/cwl_ws/cwl_sandbox/docker_mount/workflow.cwl
[workflow workflow.cwl] workflow starting
[workflow workflow.cwl] starting step files_to_folder
[job step files_to_folder] job input {
    "file:///Users/kot4or/workspaces/cwl_ws/cwl_sandbox/docker_mount/workflow.cwl#files_to_folder/input_files": [
        {
            "class": "File", 
            "location": "file:///Users/kot4or/workspaces/cwl_ws/cwl_sandbox/docker_mount/empty_file_1.txt", 
            "basename": "empty_file_1.txt"
        }, 
        {
            "class": "File", 
            "location": "file:///Users/kot4or/workspaces/cwl_ws/cwl_sandbox/docker_mount/empty_file_2.txt", 
            "basename": "empty_file_2.txt"
        }
    ]
}
[job step files_to_folder] evaluated job input to {
    "file:///Users/kot4or/workspaces/cwl_ws/cwl_sandbox/docker_mount/workflow.cwl#files_to_folder/input_files": [
        {
            "class": "File", 
            "location": "file:///Users/kot4or/workspaces/cwl_ws/cwl_sandbox/docker_mount/empty_file_1.txt", 
            "basename": "empty_file_1.txt"
        }, 
        {
            "class": "File", 
            "location": "file:///Users/kot4or/workspaces/cwl_ws/cwl_sandbox/docker_mount/empty_file_2.txt", 
            "basename": "empty_file_2.txt"
        }
    ]
}
[step files_to_folder] produced output {
    "file:///Users/kot4or/workspaces/cwl_ws/cwl_sandbox/docker_mount/workflow.cwl#files_to_folder/folder": {
        "basename": "folder", 
        "location": "_:472fab15-63fb-42fd-9c0a-1fc99830cd2c", 
        "class": "Directory", 
        "listing": [
            {
                "basename": "empty_file_1.txt", 
                "class": "File", 
                "location": "file:///Users/kot4or/workspaces/cwl_ws/cwl_sandbox/docker_mount/empty_file_1.txt"
            }, 
            {
                "basename": "empty_file_2.txt", 
                "class": "File", 
                "location": "file:///Users/kot4or/workspaces/cwl_ws/cwl_sandbox/docker_mount/empty_file_2.txt"
            }
        ]
    }
}
[step files_to_folder] completed success
[workflow workflow.cwl] starting step list_dir
[job step list_dir] job input {
    "file:///Users/kot4or/workspaces/cwl_ws/cwl_sandbox/docker_mount/workflow.cwl#list_dir/input_dir": {
        "basename": "folder", 
        "listing": [
            {
                "basename": "empty_file_1.txt", 
                "class": "File", 
                "location": "file:///Users/kot4or/workspaces/cwl_ws/cwl_sandbox/docker_mount/empty_file_1.txt"
            }, 
            {
                "basename": "empty_file_2.txt", 
                "class": "File", 
                "location": "file:///Users/kot4or/workspaces/cwl_ws/cwl_sandbox/docker_mount/empty_file_2.txt"
            }
        ], 
        "location": "_:472fab15-63fb-42fd-9c0a-1fc99830cd2c", 
        "class": "Directory"
    }
}
[job step list_dir] evaluated job input to {
    "file:///Users/kot4or/workspaces/cwl_ws/cwl_sandbox/docker_mount/workflow.cwl#list_dir/input_dir": {
        "basename": "folder", 
        "listing": [
            {
                "basename": "empty_file_1.txt", 
                "class": "File", 
                "location": "file:///Users/kot4or/workspaces/cwl_ws/cwl_sandbox/docker_mount/empty_file_1.txt"
            }, 
            {
                "basename": "empty_file_2.txt", 
                "class": "File", 
                "location": "file:///Users/kot4or/workspaces/cwl_ws/cwl_sandbox/docker_mount/empty_file_2.txt"
            }
        ], 
        "location": "_:472fab15-63fb-42fd-9c0a-1fc99830cd2c", 
        "class": "Directory"
    }
}
[job list_dir] initializing from file:///Users/kot4or/workspaces/cwl_ws/cwl_sandbox/docker_mount/list_dir.cwl as part of step list_dir
[job list_dir] {
    "input_dir": {
        "basename": "folder", 
        "listing": [
            {
                "basename": "empty_file_1.txt", 
                "class": "File", 
                "location": "file:///Users/kot4or/workspaces/cwl_ws/cwl_sandbox/docker_mount/empty_file_1.txt"
            }, 
            {
                "basename": "empty_file_2.txt", 
                "class": "File", 
                "location": "file:///Users/kot4or/workspaces/cwl_ws/cwl_sandbox/docker_mount/empty_file_2.txt"
            }
        ], 
        "location": "_:472fab15-63fb-42fd-9c0a-1fc99830cd2c", 
        "class": "Directory"
    }
}
[job list_dir] path mappings is {
    "file:///Users/kot4or/workspaces/cwl_ws/cwl_sandbox/docker_mount/empty_file_1.txt": [
        "/Users/kot4or/workspaces/cwl_ws/cwl_sandbox/docker_mount/empty_file_1.txt", 
        "/private/var/lib/cwl/stgc2e94276-c16d-4e65-98a2-d3eb02f38cf5/folder/empty_file_1.txt", 
        "File", 
        true
    ], 
    "_:472fab15-63fb-42fd-9c0a-1fc99830cd2c": [
        "_:472fab15-63fb-42fd-9c0a-1fc99830cd2c", 
        "/private/var/lib/cwl/stgc2e94276-c16d-4e65-98a2-d3eb02f38cf5/folder", 
        "Directory", 
        true
    ], 
    "file:///Users/kot4or/workspaces/cwl_ws/cwl_sandbox/docker_mount/empty_file_2.txt": [
        "/Users/kot4or/workspaces/cwl_ws/cwl_sandbox/docker_mount/empty_file_2.txt", 
        "/private/var/lib/cwl/stgc2e94276-c16d-4e65-98a2-d3eb02f38cf5/folder/empty_file_2.txt", 
        "File", 
        true
    ]
}
[job list_dir] command line bindings is [
    {
        "position": [
            -1000000, 
            0
        ], 
        "datum": "ls"
    }, 
    {
        "valueFrom": "$(inputs.input_dir)", 
        "position": [
            1, 
            0
        ]
    }
]
[job list_dir] /var/folders/sd/41rg42_16q72_2yzl_vvgsbw0000gn/T/tmphPh_f3$ docker \
    run \
    -i \
    --volume=/Users/kot4or/workspaces/cwl_ws/cwl_sandbox/docker_mount/empty_file_1.txt:/private/var/lib/cwl/stgc2e94276-c16d-4e65-98a2-d3eb02f38cf5/folder/empty_file_1.txt:ro \
    --volume=/Users/kot4or/workspaces/cwl_ws/cwl_sandbox/docker_mount/empty_file_2.txt:/private/var/lib/cwl/stgc2e94276-c16d-4e65-98a2-d3eb02f38cf5/folder/empty_file_2.txt:ro \
    --volume=_:472fab15-63fb-42fd-9c0a-1fc99830cd2c:/private/var/lib/cwl/stgc2e94276-c16d-4e65-98a2-d3eb02f38cf5/folder:ro \ 
    --volume=/private/var/folders/sd/41rg42_16q72_2yzl_vvgsbw0000gn/T/tmphPh_f3:/private/var/spool/cwl:rw \
    --volume=/private/var/folders/sd/41rg42_16q72_2yzl_vvgsbw0000gn/T/tmpLyN4Rb:/tmp:rw \
    --workdir=/private/var/spool/cwl \
    --read-only=true \
    --user=501 \
    --rm \
    --env=TMPDIR=/tmp \
    --env=HOME=/private/var/spool/cwl \
    ubuntu \
    ls \
    /private/var/lib/cwl/stgc2e94276-c16d-4e65-98a2-d3eb02f38cf5/folder
docker: Error response from daemon: invalid bind mount spec "_:472fab15-63fb-42fd-9c0a-1fc99830cd2c:/private/var/lib/cwl/stgc2e94276-c16d-4e65-98a2-d3eb02f38cf5/folder:ro": invalid volume specification: '_:472fab15-63fb-42fd-9c0a-1fc99830cd2c:/private/var/lib/cwl/stgc2e94276-c16d-4e65-98a2-d3eb02f38cf5/folder:ro'.
See 'docker run --help'.
[job list_dir] completed permanentFail
[job list_dir] {}
[step list_dir] produced output {}
[step list_dir] completed permanentFail
[job list_dir] Removing input staging directory /var/folders/sd/41rg42_16q72_2yzl_vvgsbw0000gn/T/tmpFbJ2B9
[job list_dir] Removing temporary directory /var/folders/sd/41rg42_16q72_2yzl_vvgsbw0000gn/T/tmpLyN4Rb
[job list_dir] Removing empty output directory /var/folders/sd/41rg42_16q72_2yzl_vvgsbw0000gn/T/tmphPh_f3
[workflow workflow.cwl] outdir is /var/folders/sd/41rg42_16q72_2yzl_vvgsbw0000gn/T/tmpdX4XQf
Removing intermediate output directory /var/folders/sd/41rg42_16q72_2yzl_vvgsbw0000gn/T/tmpdX4XQf
{}
Final process status is permanentFail
   
  

[tetron]

Thanks! I think you have the right idea, I'll take a look at it.

[kapilkd13]

Hi @michael-kotliar I tested this workflow and its working fine now. I guess a PR has been made for it replicating your changes here https://github.com/SciDAP/cwltool/commit/2b4eaef7fa30e40845a6308350fca5d67517b5e8 Can you confirm that it works for you too. We can close this issue then 😄