enable Windows CPU CI on GHA #7475

pmeier · 2023-03-30T08:55:12Z

Same deal as for the other migrations here: let's run the CircleCI and GHA tests in parallel for a few weeks and if nothing comes up, we can remove the ones on CircleCI. This only ports the CPU jobs for now due to pytorch/test-infra#3979.

cc @seemethere

pytorch-bot · 2023-03-30T08:55:16Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/vision/7475

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 Failures

As of commit c5c129f:

NEW FAILURES - The following jobs have failed:

unittests (3.8, macos-m1-12) / macos-job (gh)

This comment was automatically generated by Dr. CI and updates every 15 minutes.

pmeier · 2023-03-30T14:41:27Z

.github/scripts/setup-env.sh

 if [[ "${OS_TYPE}" == "macos" && $(uname -m) == x86_64 ]]; then
+  echo '::group::Uninstall system JPEG libraries on macOS'


This just moves the grouping inside the if so this doesn't show up empty on non-macos runners.

pmeier · 2023-03-30T14:45:08Z

.github/workflows/test-windows.yml

+        # FIXME: Port this to pytorch/test-infra/.github/workflows/windows_job.yml
+        export PATH="/c/Jenkins/Miniconda3/Scripts:${PATH}"


I need to dig why putting this into the windows job template as I did in pytorch/test-infra#3960 didn't do the job.

In any case, since this is just fixing the PATH for the conda binary, this is not blocking for this PR and we can fix later.

This reverts commit 44b1c0b.

This reverts commit c9b3727.

This reverts commit 4a3a49c.

pmeier · 2023-04-03T10:25:10Z

.github/scripts/setup-env.sh

@@ -66,10 +69,24 @@ ltt install --progress-bar=off \
  torch

 if [[ $GPU_ARCH_TYPE == 'cuda' ]]; then
-  python3 -c "import torch; exit(not torch.cuda.is_available())"
+  python -c "import torch; exit(not torch.cuda.is_available())"


python3 is not not available in conda envs on Windows.

pmeier · 2023-04-03T10:26:12Z

.github/scripts/unittest.sh

@@ -4,15 +4,11 @@ set -euo pipefail

 ./.github/scripts/setup-env.sh

-# Prepare conda
-CONDA_PATH=$(which conda)


Shorten this a little. No functional difference, but can be easier copy-pasted during SSH debug sessions in CI.

pmeier · 2023-04-03T10:26:50Z

.github/scripts/unittest.sh


 echo '::group::Install testing utilities'
 pip install --progress-bar=off pytest pytest-mock pytest-cov
 echo '::endgroup::'

-echo '::group::Run unittests'


After a command is finished, GH automatically collapses groups. Since we want to see the tests, drop the grouping here.

atalman

LGTM

pmeier · 2023-04-04T12:06:37Z

@atalman Just downloading nvjpeg from conda didn't work since we only look for it in $CUDA_HOME:

vision/setup.py

Lines 313 to 317 in 781f512

    
           nvjpeg_found = ( 
        
               extension is CUDAExtension 
        
               and CUDA_HOME is not None 
        
               and os.path.exists(os.path.join(CUDA_HOME, "include", "nvjpeg.h")) 
        
           )

Until we rebuild the base images to include this by default, we could do the same as we do in our packaging scripts:

vision/packaging/windows/internal/cuda_install.bat

Lines 59 to 83 in 781f512

    
           :cuda117 
        
           set CUDA_INSTALL_EXE=cuda_11.7.0_516.01_windows.exe 
        
           if not exist "%SRC_DIR%\temp_build\%CUDA_INSTALL_EXE%" ( 
        
               curl -k -L "https://ossci-windows.s3.amazonaws.com/%CUDA_INSTALL_EXE%" --output "%SRC_DIR%\temp_build\%CUDA_INSTALL_EXE%" 
        
               if errorlevel 1 exit /b 1 
        
               set "CUDA_SETUP_FILE=%SRC_DIR%\temp_build\%CUDA_INSTALL_EXE%" 
        
               set "ARGS=thrust_11.7 nvcc_11.7 cuobjdump_11.7 nvprune_11.7 nvprof_11.7 cupti_11.7 cublas_11.7 cublas_dev_11.7 cudart_11.7 cufft_11.7 cufft_dev_11.7 curand_11.7 curand_dev_11.7 cusolver_11.7 cusolver_dev_11.7 cusparse_11.7 cusparse_dev_11.7 npp_11.7 npp_dev_11.7 nvjpeg_11.7 nvjpeg_dev_11.7 nvrtc_11.7 nvrtc_dev_11.7 nvml_dev_11.7" 
        
           ) 
        
           set CUDNN_INSTALL_ZIP=cudnn-windows-x86_64-8.3.2.44_cuda11.5-archive.zip 
        
           set CUDNN_FOLDER=cudnn-windows-x86_64-8.3.2.44_cuda11.5-archive 
        
           set CUDNN_LIB_FOLDER="lib" 
        
           if not exist "%SRC_DIR%\temp_build\%CUDNN_INSTALL_ZIP%" ( 
        
               curl -k -L "http://s3.amazonaws.com/ossci-windows/%CUDNN_INSTALL_ZIP%" --output "%SRC_DIR%\temp_build\%CUDNN_INSTALL_ZIP%" 
        
               if errorlevel 1 exit /b 1 
        
               set "CUDNN_SETUP_FILE=%SRC_DIR%\temp_build\%CUDNN_INSTALL_ZIP%" 
        
               rem Make sure windows path contains zlib dll 
        
               curl -k -L "http://s3.amazonaws.com/ossci-windows/zlib123dllx64.zip" --output "%SRC_DIR%\temp_build\zlib123dllx64.zip" 
        
               7z x "%SRC_DIR%\temp_build\zlib123dllx64.zip" -o"%SRC_DIR%\temp_build\zlib" 
        
               xcopy /Y "%SRC_DIR%\temp_build\zlib\dll_x64\*.dll" "C:\Windows\System32" 
        
           ) 
        
           goto cuda_common

However, due to pytorch/test-infra#3986 it is currently almost impossible for me to try this. Thus, to unblock this PR, I'm going to remove the GPU workflow for now.

When nvjpeg is available by default on the Windows runners, or they spin up predictable enough for me to SSH in and debug, we can re-enable them.

This reverts commit 1c13a08.

github-actions · 2023-04-04T14:09:58Z

Hey @pmeier!

You merged this PR, but no labels were added. The list of valid labels is available at https://github.com/pytorch/vision/blob/main/.github/process_commit.py

Reviewed By: vmoens Differential Revision: D45183662 fbshipit-source-id: 4a8562c760d3551680b5fe3ac36da3ed52e33aee

pmeier added 2 commits March 30, 2023 10:54

add minimal windows CI

756cb5c

[REVERTME] disable unrelated CI

4a3a49c

facebook-github-bot added the cla signed label Mar 30, 2023

pmeier added module: ci and removed cla signed labels Mar 30, 2023

fix

04398de

facebook-github-bot added the cla signed label Mar 30, 2023

pmeier added 8 commits March 30, 2023 11:14

move debug to actual script

4133cdc

try conda first

a5d0821

try fixing path manually

632fceb

trigger CI

5282b45

try install third party dependencies prior to TorchVision install

270638a

expand matrix

b792e1d

add comments

83cea1a

Merge branch 'main' into win-ci

e8b0651

pmeier commented Mar 30, 2023

View reviewed changes

fix indent

53a1c37

pmeier changed the title ~~[DEBUG] enable Windows CI on GHA~~ enable Windows CI on GHA Mar 30, 2023

pmeier added 10 commits March 30, 2023 17:11

fix python invocation

59b5562

remove grouping around unit tests

4cea941

switch to CUDA 11.8

4bee1f2

set CUDA_HOME and CUDA_PATH

bbf58cd

only set paths for CUDA runners

80e3605

FIXME -> TODO

751d76a

try use windows style path

44b1c0b

[REVERTME] only run smoke tests

c585d29

use bash echo mode

3788c01

Revert "try use windows style path"

a4cdf02

This reverts commit 44b1c0b.

pmeier added 10 commits March 31, 2023 11:24

Merge branch 'main' into win-ci

7227ae2

move requirement installation just before regular one

05dbfb3

fix requirement install

3e5719b

trigger CI

6d06970

[REVERTME] disable CPU workflows

c9b3727

Merge branch 'win-ci' of https://github.com/pmeier/vision into win-ci

c70d95e

add check for nvjpeg for CUDA jobs

a846096

Revert "[REVERTME] disable CPU workflows"

26838ed

This reverts commit c9b3727.

Revert "[REVERTME] disable unrelated CI"

a5a400a

This reverts commit 4a3a49c.

disable GPU tests for now

4714a7f

pmeier commented Apr 3, 2023

View reviewed changes

pmeier mentioned this pull request Apr 3, 2023

nvJPEG is missing from Windows GPU runners pytorch/test-infra#3979

Closed

pmeier changed the title ~~enable Windows CI on GHA~~ enable Windows CPU CI on GHA Apr 3, 2023

pmeier requested a review from atalman April 3, 2023 12:00

pmeier marked this pull request as ready for review April 3, 2023 12:00

pmeier mentioned this pull request Apr 3, 2023

CircleCI to GitHub Actions tracker #7405

Closed

21 tasks

Merge branch 'main' into win-ci

8ae0b19

atalman approved these changes Apr 3, 2023

View reviewed changes

pmeier added 3 commits April 3, 2023 16:58

try install nvjpeg from conda

bd94248

Merge branch 'win-ci' of https://github.com/pmeier/vision into win-ci

d4c551d

enable GPU job

1c13a08

pmeier added 2 commits April 4, 2023 14:07

Revert "enable GPU job"

866040e

This reverts commit 1c13a08.

remove conda install of nvjpeg

c5c129f

pmeier merged commit 5c5a94d into pytorch:main Apr 4, 2023

pmeier deleted the win-ci branch April 4, 2023 14:09

pmeier mentioned this pull request Apr 4, 2023

enable Windows GPU CI on GHA #7498

Merged

pmeier mentioned this pull request Apr 17, 2023

disable Windows CPU unittests on CircleCI #7526

Merged

facebook-github-bot pushed a commit that referenced this pull request Apr 24, 2023

[fbsync] enable Windows CPU CI on GHA (#7475)

e4537b0

Reviewed By: vmoens Differential Revision: D45183662 fbshipit-source-id: 4a8562c760d3551680b5fe3ac36da3ed52e33aee

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

enable Windows CPU CI on GHA #7475

enable Windows CPU CI on GHA #7475

pmeier commented Mar 30, 2023 •

edited

Loading

pytorch-bot bot commented Mar 30, 2023 •

edited

Loading

pmeier Mar 30, 2023

pmeier Mar 30, 2023

pmeier Apr 3, 2023

pmeier Apr 3, 2023

pmeier Apr 3, 2023

atalman left a comment

pmeier commented Apr 4, 2023

github-actions bot commented Apr 4, 2023

		if [[ "${OS_TYPE}" == "macos" && $(uname -m) == x86_64 ]]; then
		echo '::group::Uninstall system JPEG libraries on macOS'

		# FIXME: Port this to pytorch/test-infra/.github/workflows/windows_job.yml
		export PATH="/c/Jenkins/Miniconda3/Scripts:${PATH}"

enable Windows CPU CI on GHA #7475

enable Windows CPU CI on GHA #7475

Conversation

pmeier commented Mar 30, 2023 • edited Loading

pytorch-bot bot commented Mar 30, 2023 • edited Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/vision/7475

❌ 1 Failures

pmeier Mar 30, 2023

Choose a reason for hiding this comment

pmeier Mar 30, 2023

Choose a reason for hiding this comment

pmeier Apr 3, 2023

Choose a reason for hiding this comment

pmeier Apr 3, 2023

Choose a reason for hiding this comment

pmeier Apr 3, 2023

Choose a reason for hiding this comment

atalman left a comment

Choose a reason for hiding this comment

pmeier commented Apr 4, 2023

github-actions bot commented Apr 4, 2023

pmeier commented Mar 30, 2023 •

edited

Loading

pytorch-bot bot commented Mar 30, 2023 •

edited

Loading