-
Notifications
You must be signed in to change notification settings - Fork 1.2k
gha: enable hdfs tests #4644
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. Weβll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
gha: enable hdfs tests #4644
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
efiop
commented
Sep 30, 2020
Comment on lines
-103
to
-104
tar = tarfile.open(tar) | ||
tar.extractall(dname) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For some reason, python's tarfile sometimes throws this error. In the interest of time leaving with os.system
for now.
2020-09-30T02:02:05.7905854Z ______________________ ERROR at setup of test_cloud[hdfs] ______________________
2020-09-30T02:02:05.7907160Z [gw3] linux -- Python 3.7.9 /opt/hostedtoolcache/Python/3.7.9/x64/bin/python
2020-09-30T02:02:05.7907751Z
2020-09-30T02:02:05.7908331Z request = <FixtureRequest for <Function test_cloud[hdfs]>>
2020-09-30T02:02:05.7908849Z
2020-09-30T02:02:05.7909303Z def fill(request):
2020-09-30T02:02:05.7909825Z item = request._pyfuncitem
2020-09-30T02:02:05.7910472Z fixturenames = getattr(item, "fixturenames", None)
2020-09-30T02:02:05.7911296Z if fixturenames is None:
2020-09-30T02:02:05.7911980Z fixturenames = request.fixturenames
2020-09-30T02:02:05.7912585Z
2020-09-30T02:02:05.7913303Z if hasattr(item, 'callspec'):
2020-09-30T02:02:05.7914167Z for param, val in sorted_by_dependency(item.callspec.params, fixturenames):
2020-09-30T02:02:05.7915106Z if val is not None and is_lazy_fixture(val):
2020-09-30T02:02:05.7916179Z > item.callspec.params[param] = request.getfixturevalue(val.name)
2020-09-30T02:02:05.7916883Z
2020-09-30T02:02:05.7917832Z /opt/hostedtoolcache/Python/3.7.9/x64/lib/python3.7/site-packages/pytest_lazyfixture.py:35:
2020-09-30T02:02:05.7918778Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
2020-09-30T02:02:05.7919522Z /home/runner/work/dvc/dvc/tests/remotes/hdfs.py:109: in hadoop
2020-09-30T02:02:05.7920194Z _get(java_url, java_tar, java_home)
2020-09-30T02:02:05.7920848Z /home/runner/work/dvc/dvc/tests/remotes/hdfs.py:104: in _get
2020-09-30T02:02:05.7921514Z tar.extractall(dname)
2020-09-30T02:02:05.7922286Z /opt/hostedtoolcache/Python/3.7.9/x64/lib/python3.7/tarfile.py:2002: in extractall
2020-09-30T02:02:05.7923064Z numeric_owner=numeric_owner)
2020-09-30T02:02:05.7923816Z /opt/hostedtoolcache/Python/3.7.9/x64/lib/python3.7/tarfile.py:2044: in extract
2020-09-30T02:02:05.7924573Z numeric_owner=numeric_owner)
2020-09-30T02:02:05.7925358Z /opt/hostedtoolcache/Python/3.7.9/x64/lib/python3.7/tarfile.py:2114: in _extract_member
2020-09-30T02:02:05.7926211Z self.makefile(tarinfo, targetpath)
2020-09-30T02:02:05.7926795Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
2020-09-30T02:02:05.7927823Z
2020-09-30T02:02:05.7928403Z self = <tarfile.TarFile object at 0x7f29e8d0e4d0>
2020-09-30T02:02:05.7929393Z tarinfo = <TarInfo 'java-se-7u75-ri/LICENSE' at 0x7f29e8d8c390>
2020-09-30T02:02:05.7930509Z targetpath = '/home/runner/.cache/dvc-test/hdfs/java-se-7u75-ri/LICENSE'
2020-09-30T02:02:05.7931014Z
2020-09-30T02:02:05.7931448Z def makefile(self, tarinfo, targetpath):
2020-09-30T02:02:05.7931992Z """Make a file called targetpath.
2020-09-30T02:02:05.7932402Z """
2020-09-30T02:02:05.7932790Z source = self.fileobj
2020-09-30T02:02:05.7933326Z source.seek(tarinfo.offset_data)
2020-09-30T02:02:05.7933875Z bufsize = self.copybufsize
2020-09-30T02:02:05.7934411Z > with bltn_open(targetpath, "wb") as target:
2020-09-30T02:02:05.7935447Z E PermissionError: [Errno 13] Permission denied: '/home/runner/.cache/dvc-test/hdfs/java-se-7u75-ri/LICENSE'
2020-09-30T02:02:05.7936247Z
2020-09-30T02:02:05.7936921Z /opt/hostedtoolcache/Python/3.7.9/x64/lib/python3.7/tarfile.py:2155: PermissionError
2020-09-30T02:02:05.7937568Z ____________________ ERROR at setup of test_cloud_cli[hdfs] ____________________
2020-09-30T02:02:05.7938791Z [gw3] linux -- Python 3.7.9 /opt/hostedtoolcache/Python/3.7.9/x64/bin/python
2020-09-30T02:02:05.7939143Z
2020-09-30T02:02:05.7939546Z request = <FixtureRequest for <Function test_cloud_cli[hdfs]>>
2020-09-30T02:02:05.7939897Z
2020-09-30T02:02:05.7940137Z def fill(request):
2020-09-30T02:02:05.7940477Z item = request._pyfuncitem
2020-09-30T02:02:05.7940946Z fixturenames = getattr(item, "fixturenames", None)
2020-09-30T02:02:05.7941409Z if fixturenames is None:
2020-09-30T02:02:05.7942113Z fixturenames = request.fixturenames
2020-09-30T02:02:05.7942501Z
2020-09-30T02:02:05.7942962Z if hasattr(item, 'callspec'):
2020-09-30T02:02:05.7943552Z for param, val in sorted_by_dependency(item.callspec.params, fixturenames):
2020-09-30T02:02:05.7944181Z if val is not None and is_lazy_fixture(val):
2020-09-30T02:02:05.7944833Z > item.callspec.params[param] = request.getfixturevalue(val.name)
2020-09-30T02:02:05.7945336Z
2020-09-30T02:02:05.7946061Z /opt/hostedtoolcache/Python/3.7.9/x64/lib/python3.7/site-packages/pytest_lazyfixture.py:35:
2020-09-30T02:02:05.7946614Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
2020-09-30T02:02:05.7946985Z /home/runner/work/dvc/dvc/tests/remotes/hdfs.py:109: in hadoop
2020-09-30T02:02:05.7947429Z _get(java_url, java_tar, java_home)
2020-09-30T02:02:05.7947858Z /home/runner/work/dvc/dvc/tests/remotes/hdfs.py:104: in _get
2020-09-30T02:02:05.7948287Z tar.extractall(dname)
2020-09-30T02:02:05.7949013Z /opt/hostedtoolcache/Python/3.7.9/x64/lib/python3.7/tarfile.py:2002: in extractall
2020-09-30T02:02:05.7949568Z numeric_owner=numeric_owner)
2020-09-30T02:02:05.7950091Z /opt/hostedtoolcache/Python/3.7.9/x64/lib/python3.7/tarfile.py:2044: in extract
2020-09-30T02:02:05.7950614Z numeric_owner=numeric_owner)
2020-09-30T02:02:05.7951240Z /opt/hostedtoolcache/Python/3.7.9/x64/lib/python3.7/tarfile.py:2114: in _extract_member
2020-09-30T02:02:05.7951830Z self.makefile(tarinfo, targetpath)
2020-09-30T02:02:05.7952193Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
2020-09-30T02:02:05.7952382Z
2020-09-30T02:02:05.7952739Z self = <tarfile.TarFile object at 0x7f29e8d0e4d0>
2020-09-30T02:02:05.7953487Z tarinfo = <TarInfo 'java-se-7u75-ri/LICENSE' at 0x7f29e8d8c390>
2020-09-30T02:02:05.7954259Z targetpath = '/home/runner/.cache/dvc-test/hdfs/java-se-7u75-ri/LICENSE'
2020-09-30T02:02:05.7954650Z
2020-09-30T02:02:05.7954988Z def makefile(self, tarinfo, targetpath):
2020-09-30T02:02:05.7955428Z """Make a file called targetpath.
2020-09-30T02:02:05.7955727Z """
2020-09-30T02:02:05.7955993Z source = self.fileobj
2020-09-30T02:02:05.7956404Z source.seek(tarinfo.offset_data)
2020-09-30T02:02:05.7956847Z bufsize = self.copybufsize
2020-09-30T02:02:05.7957274Z > with bltn_open(targetpath, "wb") as target:
2020-09-30T02:02:05.7958179Z E PermissionError: [Errno 13] Permission denied: '/home/runner/.cache/dvc-test/hdfs/java-se-7u75-ri/LICENSE'
2020-09-30T02:02:05.7958693Z
2020-09-30T02:02:05.7959187Z /opt/hostedtoolcache/Python/3.7.9/x64/lib/python3.7/tarfile.py:2155: PermissionError
2020-09-30T02:02:05.7959635Z
2020-09-30T02:02:05.7960149Z ----------- coverage: platform linux, python 3.7.9-final-0 -----------
2020-09-30T02:02:05.7960623Z Coverage XML written to file coverage.xml
2020-09-30T02:02:05.7960898Z
2020-09-30T02:02:05.7961144Z ===Flaky Test Report===
2020-09-30T02:02:05.7961334Z
2020-09-30T02:02:05.7961745Z test_pull_git_imports[erepo_dir] passed 1 out of the required 1 times. Success!
2020-09-30T02:02:05.7962346Z test_pull_git_imports[git_dir] passed 1 out of the required 1 times. Success!
2020-09-30T02:02:05.7962701Z
2020-09-30T02:02:05.7962943Z ===End Flaky Test Report===
2020-09-30T02:02:05.7963285Z ============================== slowest durations ===============================
2020-09-30T02:02:05.7964844Z 114.47s setup tests/func/test_add.py::test_add_external_file[hdfs-checksum-000002000000000000000000a86fe4d846edc1bf4c355cb6112f141e]
2020-09-30T02:02:05.7965963Z 107.88s setup tests/func/test_api.py::test_get_url[hdfs]
2020-09-30T02:02:05.7966406Z 94.24s setup tests/func/test_api.py::test_get_url[oss]
2020-09-30T02:02:05.7966879Z 84.56s setup tests/func/test_api.py::test_open_granular[azure]
2020-09-30T02:02:05.7967403Z 25.33s call tests/func/test_update.py::test_update_import_url[hdfs]
2020-09-30T02:02:05.7969991Z 21.79s call tests/func/test_run_multistage.py::test_run_external_outputs[hdfs-checksum-0000020000000000000000003dba826b9be9c6a8e2f8310a770555c4-00000200000000000000000075433c81259d3c38e364b348af52e84d]
2020-09-30T02:02:05.7974006Z 20.01s call tests/func/test_add.py::test_add_external_file[hdfs-checksum-
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Part of #4418
β I have followed the Contributing to DVC checklist.
π If this PR requires documentation updates, I have created a separate PR (or issue, at least) in dvc.org and linked it here.
Thank you for the contribution - we'll try to review it as soon as possible. π