Skip to content

gha: enable hdfs tests #4644

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Sep 30, 2020
Merged

gha: enable hdfs tests #4644

merged 1 commit into from
Sep 30, 2020

Conversation

efiop
Copy link
Contributor

@efiop efiop commented Sep 30, 2020

Part of #4418

Thank you for the contribution - we'll try to review it as soon as possible. πŸ™

Comment on lines -103 to -104
tar = tarfile.open(tar)
tar.extractall(dname)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For some reason, python's tarfile sometimes throws this error. In the interest of time leaving with os.system for now.

2020-09-30T02:02:05.7905854Z ______________________ ERROR at setup of test_cloud[hdfs] ______________________
2020-09-30T02:02:05.7907160Z [gw3] linux -- Python 3.7.9 /opt/hostedtoolcache/Python/3.7.9/x64/bin/python
2020-09-30T02:02:05.7907751Z 
2020-09-30T02:02:05.7908331Z request = <FixtureRequest for <Function test_cloud[hdfs]>>
2020-09-30T02:02:05.7908849Z 
2020-09-30T02:02:05.7909303Z     def fill(request):
2020-09-30T02:02:05.7909825Z         item = request._pyfuncitem
2020-09-30T02:02:05.7910472Z         fixturenames = getattr(item, "fixturenames", None)
2020-09-30T02:02:05.7911296Z         if fixturenames is None:
2020-09-30T02:02:05.7911980Z             fixturenames = request.fixturenames
2020-09-30T02:02:05.7912585Z     
2020-09-30T02:02:05.7913303Z         if hasattr(item, 'callspec'):
2020-09-30T02:02:05.7914167Z             for param, val in sorted_by_dependency(item.callspec.params, fixturenames):
2020-09-30T02:02:05.7915106Z                 if val is not None and is_lazy_fixture(val):
2020-09-30T02:02:05.7916179Z >                   item.callspec.params[param] = request.getfixturevalue(val.name)
2020-09-30T02:02:05.7916883Z 
2020-09-30T02:02:05.7917832Z /opt/hostedtoolcache/Python/3.7.9/x64/lib/python3.7/site-packages/pytest_lazyfixture.py:35: 
2020-09-30T02:02:05.7918778Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
2020-09-30T02:02:05.7919522Z /home/runner/work/dvc/dvc/tests/remotes/hdfs.py:109: in hadoop
2020-09-30T02:02:05.7920194Z     _get(java_url, java_tar, java_home)
2020-09-30T02:02:05.7920848Z /home/runner/work/dvc/dvc/tests/remotes/hdfs.py:104: in _get
2020-09-30T02:02:05.7921514Z     tar.extractall(dname)
2020-09-30T02:02:05.7922286Z /opt/hostedtoolcache/Python/3.7.9/x64/lib/python3.7/tarfile.py:2002: in extractall
2020-09-30T02:02:05.7923064Z     numeric_owner=numeric_owner)
2020-09-30T02:02:05.7923816Z /opt/hostedtoolcache/Python/3.7.9/x64/lib/python3.7/tarfile.py:2044: in extract
2020-09-30T02:02:05.7924573Z     numeric_owner=numeric_owner)
2020-09-30T02:02:05.7925358Z /opt/hostedtoolcache/Python/3.7.9/x64/lib/python3.7/tarfile.py:2114: in _extract_member
2020-09-30T02:02:05.7926211Z     self.makefile(tarinfo, targetpath)
2020-09-30T02:02:05.7926795Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
2020-09-30T02:02:05.7927823Z 
2020-09-30T02:02:05.7928403Z self = <tarfile.TarFile object at 0x7f29e8d0e4d0>
2020-09-30T02:02:05.7929393Z tarinfo = <TarInfo 'java-se-7u75-ri/LICENSE' at 0x7f29e8d8c390>
2020-09-30T02:02:05.7930509Z targetpath = '/home/runner/.cache/dvc-test/hdfs/java-se-7u75-ri/LICENSE'
2020-09-30T02:02:05.7931014Z 
2020-09-30T02:02:05.7931448Z     def makefile(self, tarinfo, targetpath):
2020-09-30T02:02:05.7931992Z         """Make a file called targetpath.
2020-09-30T02:02:05.7932402Z         """
2020-09-30T02:02:05.7932790Z         source = self.fileobj
2020-09-30T02:02:05.7933326Z         source.seek(tarinfo.offset_data)
2020-09-30T02:02:05.7933875Z         bufsize = self.copybufsize
2020-09-30T02:02:05.7934411Z >       with bltn_open(targetpath, "wb") as target:
2020-09-30T02:02:05.7935447Z E       PermissionError: [Errno 13] Permission denied: '/home/runner/.cache/dvc-test/hdfs/java-se-7u75-ri/LICENSE'
2020-09-30T02:02:05.7936247Z 
2020-09-30T02:02:05.7936921Z /opt/hostedtoolcache/Python/3.7.9/x64/lib/python3.7/tarfile.py:2155: PermissionError
2020-09-30T02:02:05.7937568Z ____________________ ERROR at setup of test_cloud_cli[hdfs] ____________________
2020-09-30T02:02:05.7938791Z [gw3] linux -- Python 3.7.9 /opt/hostedtoolcache/Python/3.7.9/x64/bin/python
2020-09-30T02:02:05.7939143Z 
2020-09-30T02:02:05.7939546Z request = <FixtureRequest for <Function test_cloud_cli[hdfs]>>
2020-09-30T02:02:05.7939897Z 
2020-09-30T02:02:05.7940137Z     def fill(request):
2020-09-30T02:02:05.7940477Z         item = request._pyfuncitem
2020-09-30T02:02:05.7940946Z         fixturenames = getattr(item, "fixturenames", None)
2020-09-30T02:02:05.7941409Z         if fixturenames is None:
2020-09-30T02:02:05.7942113Z             fixturenames = request.fixturenames
2020-09-30T02:02:05.7942501Z     
2020-09-30T02:02:05.7942962Z         if hasattr(item, 'callspec'):
2020-09-30T02:02:05.7943552Z             for param, val in sorted_by_dependency(item.callspec.params, fixturenames):
2020-09-30T02:02:05.7944181Z                 if val is not None and is_lazy_fixture(val):
2020-09-30T02:02:05.7944833Z >                   item.callspec.params[param] = request.getfixturevalue(val.name)
2020-09-30T02:02:05.7945336Z 
2020-09-30T02:02:05.7946061Z /opt/hostedtoolcache/Python/3.7.9/x64/lib/python3.7/site-packages/pytest_lazyfixture.py:35: 
2020-09-30T02:02:05.7946614Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
2020-09-30T02:02:05.7946985Z /home/runner/work/dvc/dvc/tests/remotes/hdfs.py:109: in hadoop
2020-09-30T02:02:05.7947429Z     _get(java_url, java_tar, java_home)
2020-09-30T02:02:05.7947858Z /home/runner/work/dvc/dvc/tests/remotes/hdfs.py:104: in _get
2020-09-30T02:02:05.7948287Z     tar.extractall(dname)
2020-09-30T02:02:05.7949013Z /opt/hostedtoolcache/Python/3.7.9/x64/lib/python3.7/tarfile.py:2002: in extractall
2020-09-30T02:02:05.7949568Z     numeric_owner=numeric_owner)
2020-09-30T02:02:05.7950091Z /opt/hostedtoolcache/Python/3.7.9/x64/lib/python3.7/tarfile.py:2044: in extract
2020-09-30T02:02:05.7950614Z     numeric_owner=numeric_owner)
2020-09-30T02:02:05.7951240Z /opt/hostedtoolcache/Python/3.7.9/x64/lib/python3.7/tarfile.py:2114: in _extract_member
2020-09-30T02:02:05.7951830Z     self.makefile(tarinfo, targetpath)
2020-09-30T02:02:05.7952193Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
2020-09-30T02:02:05.7952382Z 
2020-09-30T02:02:05.7952739Z self = <tarfile.TarFile object at 0x7f29e8d0e4d0>
2020-09-30T02:02:05.7953487Z tarinfo = <TarInfo 'java-se-7u75-ri/LICENSE' at 0x7f29e8d8c390>
2020-09-30T02:02:05.7954259Z targetpath = '/home/runner/.cache/dvc-test/hdfs/java-se-7u75-ri/LICENSE'
2020-09-30T02:02:05.7954650Z 
2020-09-30T02:02:05.7954988Z     def makefile(self, tarinfo, targetpath):
2020-09-30T02:02:05.7955428Z         """Make a file called targetpath.
2020-09-30T02:02:05.7955727Z         """
2020-09-30T02:02:05.7955993Z         source = self.fileobj
2020-09-30T02:02:05.7956404Z         source.seek(tarinfo.offset_data)
2020-09-30T02:02:05.7956847Z         bufsize = self.copybufsize
2020-09-30T02:02:05.7957274Z >       with bltn_open(targetpath, "wb") as target:
2020-09-30T02:02:05.7958179Z E       PermissionError: [Errno 13] Permission denied: '/home/runner/.cache/dvc-test/hdfs/java-se-7u75-ri/LICENSE'
2020-09-30T02:02:05.7958693Z 
2020-09-30T02:02:05.7959187Z /opt/hostedtoolcache/Python/3.7.9/x64/lib/python3.7/tarfile.py:2155: PermissionError
2020-09-30T02:02:05.7959635Z 
2020-09-30T02:02:05.7960149Z ----------- coverage: platform linux, python 3.7.9-final-0 -----------
2020-09-30T02:02:05.7960623Z Coverage XML written to file coverage.xml
2020-09-30T02:02:05.7960898Z 
2020-09-30T02:02:05.7961144Z ===Flaky Test Report===
2020-09-30T02:02:05.7961334Z 
2020-09-30T02:02:05.7961745Z test_pull_git_imports[erepo_dir] passed 1 out of the required 1 times. Success!
2020-09-30T02:02:05.7962346Z test_pull_git_imports[git_dir] passed 1 out of the required 1 times. Success!
2020-09-30T02:02:05.7962701Z 
2020-09-30T02:02:05.7962943Z ===End Flaky Test Report===
2020-09-30T02:02:05.7963285Z ============================== slowest durations ===============================
2020-09-30T02:02:05.7964844Z 114.47s setup    tests/func/test_add.py::test_add_external_file[hdfs-checksum-000002000000000000000000a86fe4d846edc1bf4c355cb6112f141e]
2020-09-30T02:02:05.7965963Z 107.88s setup    tests/func/test_api.py::test_get_url[hdfs]
2020-09-30T02:02:05.7966406Z 94.24s setup    tests/func/test_api.py::test_get_url[oss]
2020-09-30T02:02:05.7966879Z 84.56s setup    tests/func/test_api.py::test_open_granular[azure]
2020-09-30T02:02:05.7967403Z 25.33s call     tests/func/test_update.py::test_update_import_url[hdfs]
2020-09-30T02:02:05.7969991Z 21.79s call     tests/func/test_run_multistage.py::test_run_external_outputs[hdfs-checksum-0000020000000000000000003dba826b9be9c6a8e2f8310a770555c4-00000200000000000000000075433c81259d3c38e364b348af52e84d]
2020-09-30T02:02:05.7974006Z 20.01s call     tests/func/test_add.py::test_add_external_file[hdfs-checksum-

@efiop efiop changed the title [WIP] gha: enable hdfs tests gha: enable hdfs tests Sep 30, 2020
@efiop efiop added ci testing Related to the tests and the testing infrastructure labels Sep 30, 2020
@efiop efiop merged commit d7a35b9 into iterative:master Sep 30, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
testing Related to the tests and the testing infrastructure
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant