[WIP] Fix windows installation error due to absence of \tmp folder #125

manu-chroma · 2017-07-31T19:33:21Z

Fixes common-workflow-language/cwltool#499
Closes common-workflow-language/cwltool#504

manu-chroma · 2017-08-01T03:53:08Z

schema_salad/__init__.py

+        # required by autotranslate module
+        if not os.path.exists("/tmp"):
+            try:
+                os.mkdirs("/tmp")


@kapilkd13 Does this mean we are checking and creating in the current directory or Drive:\ root? Also, can you test this code?

kapilkd13 · 2017-08-02T20:30:44Z

Hi @manu-chroma I tried this PR on python 3 and I got this error

E:\GitHub\gsoc>cwltool
Traceback (most recent call last):
  File "C:\python36\Scripts\cwltool-script.py", line 11, in <module>
    load_entry_point('cwltool==1.0.20170801175120', 'console_scripts', 'cwltool'
)()
  File "C:\python36\lib\site-packages\pkg_resources\__init__.py", line 565, in l
oad_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "C:\python36\lib\site-packages\pkg_resources\__init__.py", line 2631, in
load_entry_point
    return ep.load()
  File "C:\python36\lib\site-packages\pkg_resources\__init__.py", line 2291, in
load
    return self.resolve()
  File "C:\python36\lib\site-packages\pkg_resources\__init__.py", line 2297, in
resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load
  File "<frozen importlib._bootstrap>", line 950, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 646, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 616, in _load_backward_compatible
  File "C:\python36\lib\site-packages\cwltool-1.0.20170801175120-py3.6.egg\cwlto
ol\main.py", line 22, in <module>
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load
  File "<frozen importlib._bootstrap>", line 950, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 646, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 616, in _load_backward_compatible
  File "C:\python36\lib\site-packages\schema_salad-2.6.20170801040821-py3.6.egg\
schema_salad\validate.py", line 3, in <module>
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load
  File "<frozen importlib._bootstrap>", line 946, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 885, in _find_spec
  File "<frozen importlib._bootstrap_external>", line 1157, in find_spec
  File "<frozen importlib._bootstrap_external>", line 1131, in _get_spec
  File "<frozen importlib._bootstrap_external>", line 1112, in _legacy_get_spec
  File "<frozen importlib._bootstrap>", line 427, in spec_from_loader
  File "<frozen importlib._bootstrap_external>", line 544, in spec_from_file_loc
ation
  File "C:\python36\lib\site-packages\avro_cwl-1.8.3-py3.6.egg\avro\schema.py",
line 341
    except Exception, e:
                    ^
SyntaxError: invalid syntax

It looks like this format of exception has been removed from py 3.
Coming to reason for this exception, looks like there is some error in avro-cwl package.
Curiously, When I removed avro-cwl egg file and pip install avro-cwl, cwltool starts correctly. I guess marking avro-cwl as not-zip-safe may help us here.
There was one other fatal error in unit tests causing 2 tests to fail

_______________________ TestFactory.test_partial_output _______________________

self = <tests.test_examples.TestFactory testMethod=test_partial_output>

    def test_partial_output(self):
        f = cwltool.factory.Factory(on_error="continue")
        fail = f.make(get_data("tests/wf/wffail.cwl"))
        try:
>           fail()

tests\test_examples.py:139:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <cwltool.factory.Callable object at 0x0000000005B1F048>, kwargs = {}
execkwargs = {'basedir': 'E:\\GitHub\\gsoc\\cwltool', 'on_error': 'continue'}
out = {'out1': None, 'out2': None, 'out4': None}, status = 'permanentFail'

    def __call__(self, **kwargs):
        # type: (**Any) -> Union[Text, Dict[Text, Text]]
        execkwargs = self.factory.execkwargs.copy()
        execkwargs["basedir"] = os.getcwd()
        out, status = self.factory.executor(self.t, kwargs, **execkwargs)
        if status != "success":
>           raise WorkflowStatus(out, status)
E           cwltool.factory.WorkflowStatus: Completed permanentFail

cwltool\factory.py:29: WorkflowStatus

During handling of the above exception, another exception occurred:

self = <tests.test_examples.TestFactory testMethod=test_partial_output>

    def test_partial_output(self):
        f = cwltool.factory.Factory(on_error="continue")
        fail = f.make(get_data("tests/wf/wffail.cwl"))
        try:
            fail()
        except cwltool.factory.WorkflowStatus as e:
>           self.assertEquals('sha1$e5fa44f2b31c1fb553b6021e7360d07d5d91ff5e', e
.out["out1"]["checksum"])
E           TypeError: 'NoneType' object is not subscriptable

tests\test_examples.py:141: TypeError
---------------------------- Captured stderr call -----------------------------
Resolved 'e:\github\gsoc\cwltool\tests\wf\wffail.cwl' to 'file:///E:/github/gsoc
/cwltool/tests/wf/wffail.cwl'
[workflow wffail.cwl] start
[step step2] start
[job step2] C:\Users\KAPILK~1\AppData\Local\Temp\tmpyc4y_w3e$ python \
    -c \
    'from __future__ import print_function
import sys
print(sys.argv[1])
if sys.argv[1] == "2":
  exit(1)
else:
  f = open("foo"+sys.argv[1]+".txt", "wb")
  content = sys.argv[1]+"\n"
  f.write(content.encode('"'"'utf-8'"'"'))
if sys.argv[1] == "5":
  exit(1)
' \
    2
Fatal Python error: Failed to initialize Windows random API (CryptoGen)
[job step2] Job error:
E:github\gsoc\cwltool\tests\wf\echo.cwl:20:3: Error collecting output for parame
ter 'out':
E:github\gsoc\cwltool\tests\wf\echo.cwl:23:7:   Did not find output file with gl
ob pattern: '['foo2.txt']'
[job step2] completed permanentFail
[step step2] Output is missing expected field file:///E:/github/gsoc/cwltool/tes
ts/wf/wffail.cwl#step2/out
[step step2] completed permanentFail
[step step1] start
[job step1] C:\Users\KAPILK~1\AppData\Local\Temp\tmpaim0v9ag$ python \
    -c \
    'from __future__ import print_function
import sys
print(sys.argv[1])
if sys.argv[1] == "2":
  exit(1)
else:
  f = open("foo"+sys.argv[1]+".txt", "wb")
  content = sys.argv[1]+"\n"
  f.write(content.encode('"'"'utf-8'"'"'))
if sys.argv[1] == "5":
  exit(1)
' \
    1
Fatal Python error: Failed to initialize Windows random API (CryptoGen)
[job step1] Job error:
E:github\gsoc\cwltool\tests\wf\echo.cwl:20:3: Error collecting output for parame
ter 'out':
E:github\gsoc\cwltool\tests\wf\echo.cwl:23:7:   Did not find output file with gl
ob pattern: '['foo1.txt']'
[job step1] completed permanentFail
[step step1] Output is missing expected field file:///E:/github/gsoc/cwltool/tes
ts/wf/wffail.cwl#step1/out
[step step1] completed permanentFail
[step step3] start
[job step3] C:\Users\KAPILK~1\AppData\Local\Temp\tmpq4bx6tij$ python \
    -c \
    'from __future__ import print_function
import sys
print(sys.argv[1])
if sys.argv[1] == "2":
  exit(1)
else:
  f = open("foo"+sys.argv[1]+".txt", "wb")
  content = sys.argv[1]+"\n"
  f.write(content.encode('"'"'utf-8'"'"'))
if sys.argv[1] == "5":
  exit(1)
' \
    5
Fatal Python error: Failed to initialize Windows random API (CryptoGen)
[job step3] Job error:
E:github\gsoc\cwltool\tests\wf\echo.cwl:20:3: Error collecting output for parame
ter 'out':
E:github\gsoc\cwltool\tests\wf\echo.cwl:23:7:   Did not find output file with gl
ob pattern: '['foo5.txt']'
[job step3] completed permanentFail
[step step3] Output is missing expected field file:///E:/github/gsoc/cwltool/tes
ts/wf/wffail.cwl#step3/out
[step step3] completed permanentFail
[workflow wffail.cwl] completed permanentFail

Here we are having Fatal Python error: Failed to initialize Windows random API (CryptoGen). On googling http://bugs.python.org/issue20614 I found that this error may occur when Systemroot env is not set. I tried setting that too, but it still fails.

kapilkd13 · 2017-08-02T20:39:01Z

@manu-chroma Also have a look at common-workflow-language/cwltool#510 There was a mistake due to which py3 tests were falling back to py2.

manu-chroma · 2017-08-02T20:39:19Z

Hey @kapilkd13

I tried this PR on python 3

Does master branch work correctly under Python 3? If yes, I will try to reproduce the error in my machine and fix the PR accordingly.

kapilkd13 · 2017-08-02T20:50:13Z

@manu-chroma Oh no. This PR is perfectly fine and solve the issue it was meant for. I just want to tell that there are still some issues before we have python 3 support on windows.

kapilkd13 · 2017-08-04T09:38:53Z

@manu-chroma This PR looks good to me. Do you want to change something here or is it ready to merge?

manu-chroma · 2017-08-04T16:16:22Z

@kapilkd13 Can you test the changes locally again? I'm using logger instead of print statement now. Thanks!

mr-c · 2017-08-06T15:59:11Z

Lets confirm and merge ASAP -- the upstream reporting checklist has been moved to #126

manu-chroma · 2017-08-06T16:04:08Z

It LGTM, just waiting for @kapilkd13 to confirm the changes from his side.

kapilkd13 · 2017-08-06T16:24:43Z

Sorry, got little late. @manu-chroma I tested with the latest changes and it is working fine. Thanks for the PR.

manu-chroma added 2 commits August 1, 2017 00:59

utils.py: add onWindows() function

f61676e

schema_salad/__init__.py: Create '\tmp' folder on windows if not present

a81a35f

manu-chroma mentioned this pull request Jul 31, 2017

removing python 3 status from windows documentation common-workflow-language/cwltool#504

Closed

manu-chroma commented Aug 1, 2017

View reviewed changes

manu-chroma force-pushed the tmp_fix_windows branch from d042a91 to 7b736bd Compare August 1, 2017 03:55

manu-chroma added 2 commits August 1, 2017 09:38

add error message when not able to create \tmp folder in Windows

ce49d5d

utils.py: Fix missing import

36c04d1

manu-chroma force-pushed the tmp_fix_windows branch from 7b736bd to 36c04d1 Compare August 1, 2017 04:08

__init__.py: Use logger to raise error on Windows rather than print

25990c5

manu-chroma force-pushed the tmp_fix_windows branch from e9b0626 to 25990c5 Compare August 4, 2017 16:14

kapilkd13 mentioned this pull request Aug 6, 2017

Python 3 support on Windows common-workflow-language/cwltool#511

Merged

mr-c requested a review from kapilkd13 August 6, 2017 13:11

kapilkd13 approved these changes Aug 6, 2017

View reviewed changes

mr-c merged commit eac115d into master Aug 6, 2017

mr-c deleted the tmp_fix_windows branch August 6, 2017 16:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Fix windows installation error due to absence of \tmp folder #125

[WIP] Fix windows installation error due to absence of \tmp folder #125

manu-chroma commented Jul 31, 2017

manu-chroma Aug 1, 2017 •

edited

Loading

kapilkd13 commented Aug 2, 2017

kapilkd13 commented Aug 2, 2017

manu-chroma commented Aug 2, 2017

kapilkd13 commented Aug 2, 2017

kapilkd13 commented Aug 4, 2017

manu-chroma commented Aug 4, 2017

mr-c commented Aug 6, 2017

manu-chroma commented Aug 6, 2017

kapilkd13 commented Aug 6, 2017

[WIP] Fix windows installation error due to absence of \tmp folder #125

[WIP] Fix windows installation error due to absence of \tmp folder #125

Conversation

manu-chroma commented Jul 31, 2017

manu-chroma Aug 1, 2017 • edited Loading

Choose a reason for hiding this comment

kapilkd13 commented Aug 2, 2017

kapilkd13 commented Aug 2, 2017

manu-chroma commented Aug 2, 2017

kapilkd13 commented Aug 2, 2017

kapilkd13 commented Aug 4, 2017

manu-chroma commented Aug 4, 2017

mr-c commented Aug 6, 2017

manu-chroma commented Aug 6, 2017

kapilkd13 commented Aug 6, 2017

manu-chroma Aug 1, 2017 •

edited

Loading