Skip to content

Fix DTD prototype dataset on Windows #5227

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 10 commits into from
Jan 31, 2022
Merged

Conversation

pmeier
Copy link
Collaborator

@pmeier pmeier commented Jan 20, 2022

This is the second time we are alerted by the torchdata folks that something broke, but without our CI is picking up on it. Both times it was related to Windows and we didn't notice, because we only run the prototype tests on a Linux machine.

I'm wondering if we should switch to a Windows machine for prototype tests in CI given that AFAIK no one of us is developing on Windows. Thoughts @datumbox @NicolasHug ?

cc @pmeier @bjuncek

@facebook-github-bot
Copy link

facebook-github-bot commented Jan 20, 2022

💊 CI failures summary and remediations

As of commit 3159f66 (more details on the Dr. CI page):


  • 2/2 failures introduced in this PR

🕵️ 1 new failure recognized by patterns

The following CI failures do not appear to be due to upstream breakages:

See GitHub Actions build CodeQL / build (1/1)

Step: "Build TorchVision" (full log | diagnosis details | 🔁 rerun)

2022-01-31T17:01:16.1616459Z ##[error]Process completed with exit code 1.
2022-01-31T17:01:16.0010808Z     self.finalize_options()
2022-01-31T17:01:16.0011288Z   File "/home/runner/.local/lib/python3.8/site-packages/setuptools/command/develop.py", line 52, in finalize_options
2022-01-31T17:01:16.0011776Z     easy_install.finalize_options(self)
2022-01-31T17:01:16.0012644Z   File "/home/runner/.local/lib/python3.8/site-packages/setuptools/command/easy_install.py", line 276, in finalize_options
2022-01-31T17:01:16.0013022Z     self._fix_install_dir_for_user_site()
2022-01-31T17:01:16.0013560Z   File "/home/runner/.local/lib/python3.8/site-packages/setuptools/command/easy_install.py", line 382, in _fix_install_dir_for_user_site
2022-01-31T17:01:16.0013933Z     self.create_home_path()
2022-01-31T17:01:16.0014676Z   File "/home/runner/.local/lib/python3.8/site-packages/setuptools/command/easy_install.py", line 1338, in create_home_path
2022-01-31T17:01:16.0015069Z     if path.startswith(home) and not os.path.isdir(path):
2022-01-31T17:01:16.0015489Z AttributeError: 'int' object has no attribute 'startswith'
2022-01-31T17:01:16.1616459Z ##[error]Process completed with exit code 1.
2022-01-31T17:01:16.1673198Z Post job cleanup.
2022-01-31T17:01:16.3166077Z [command]/usr/bin/git version
2022-01-31T17:01:16.3228584Z git version 2.34.1
2022-01-31T17:01:16.3279658Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2022-01-31T17:01:16.3338330Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :
2022-01-31T17:01:16.3762140Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2022-01-31T17:01:16.3814274Z http.https://github.com/.extraheader
2022-01-31T17:01:16.3829279Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader
2022-01-31T17:01:16.3892289Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :
2022-01-31T17:01:16.4547416Z Cleaning up orphan processes

1 failure not recognized by patterns:

Job Step Action
CircleCI cmake_macos_cpu curl -o conda.sh https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-x86_64.sh
sh conda.sh -b
source $HOME/miniconda3/bin/activate
conda install -yq conda-build cmake
packaging/build_cmake.sh
🔁 rerun

This comment was automatically generated by Dr. CI (expand for details).

Please report bugs/suggestions to the (internal) Dr. CI Users group.

Click here to manually regenerate this comment.

@pmeier pmeier changed the title [DEBUG] run prototype tests on Windows Fix line endings in prototype datasets mock data generation Jan 20, 2022
@pmeier pmeier requested a review from NicolasHug January 20, 2022 08:52
@pmeier pmeier marked this pull request as ready for review January 20, 2022 08:52
@pmeier
Copy link
Collaborator Author

pmeier commented Jan 20, 2022

Turns out, this is actually an issue in torchdata with properly stripping the newline characters: pytorch/data#173.

@pmeier pmeier marked this pull request as draft January 20, 2022 09:28
@pmeier
Copy link
Collaborator Author

pmeier commented Jan 20, 2022

With my fix in pytorch/data#174, we do not need to care about the line endings while generating or reading the files. Thus, this PR only fixes the mock data generation of DTD.

@pmeier pmeier changed the title Fix line endings in prototype datasets mock data generation Fix DTD prototype mock data generation Jan 20, 2022
@pmeier pmeier marked this pull request as ready for review January 20, 2022 09:44
@pmeier pmeier changed the title Fix DTD prototype mock data generation Fix DTD prototype dataset on Windows Jan 31, 2022
Copy link
Member

@NicolasHug NicolasHug left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @pmeier

@pmeier pmeier merged commit 8aad0e0 into pytorch:main Jan 31, 2022
@pmeier pmeier deleted the datasets/windows-ci branch January 31, 2022 17:18
facebook-github-bot pushed a commit that referenced this pull request Feb 3, 2022
Summary:
* [DEBUG] run prototype tests on Windows

* disable duplicate resource class

* fix newlines in mock data generation

* fix path matching for DTD

* revert CI changes

* Revert "fix newlines in mock data generation"

This reverts commit f53cf60.

* rerun tests with fix in torchdata

* revert CI changes

* add explanation

Reviewed By: kazhang

Differential Revision: D33927486

fbshipit-source-id: 6242128b812f2c797b9c8b69d3d7b5c9bf34fd0e
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants