Skip to content

reduce GHA log output #7267

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Feb 22, 2023
Merged

reduce GHA log output #7267

merged 4 commits into from
Feb 22, 2023

Conversation

pmeier
Copy link
Collaborator

@pmeier pmeier commented Feb 16, 2023

Currently opening the logs of a GHA unittest job is really sluggish compared to CircleCI. The reason for this is that in the new workflows we only have a single step rather than multiple ones. Meaning, while CircleCI just needs to load a portion of the total output, GHA needs to load everything. (and that is leaving out that GHA in general seems to be a lot slower at that).

While not a fix, this PR addresses this issue:

  1. There is no need for install progress to be logged in a non-interactive environment. Especially progress bars create an enormous amount of lines without any real benefit. Thus, we set the respective flags for conda and pip to install "quietly" without touching the meaningful information.
  2. Our test are run with the -v flag. This means, every test creates a single line in the log output, whereas without we only get a single character per test. Thus depending on the console width (I think it is 80 in CI), we create roughly 60-80x more lines than necessary. In some cases GHA even truncates the displayed logs (reduce verbosity of prototype tests and split them into multiple steps #6587), so one has to dig into the raw ones, which are harder to parse. On CircleCI this is not as big of a problem since they have the nifty "Tests" tab that show all failed tests without going manually through the logs.

cc @seemethere

@pmeier
Copy link
Collaborator Author

pmeier commented Feb 16, 2023

@NicolasHug
Copy link
Member

I'd really prefer keeping that kind of verboe pytest output if we can

image

@@ -39,7 +39,7 @@ jobs:
fi

# Create Conda Env
conda create -yp ci_env python="${PYTHON_VERSION}" numpy libpng jpeg scipy
conda create -yp ci_env --quiet python="${PYTHON_VERSION}" numpy libpng jpeg scipy
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is going to be very frustrating when we have env issues. So... once a week?
😔😔😔😔

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This only affects the progress bar:

$ conda create --help | grep -e --quiet
  -q, --quiet           Do not display progress bar.

The actual information about the environment, i.e what is installed / upgraded / downgraded is still there: https://github.com/pytorch/vision/actions/runs/4194550278/jobs/7272863693#step:10:168

Copy link
Collaborator

@vfdev-5 vfdev-5 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @pmeier !

Copy link
Member

@NicolasHug NicolasHug left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Even though I'd really prefer not to, I'll approve because on top of GA being frustratingly slow, it has now started to truncate the test logs as in https://github.com/pytorch/vision/actions/runs/4234487154/jobs/7356840736. This makes the tests logs impossible to see without additional clicks + we lose all formatting which makes them very hard to read.

@seemethere @osalpekar please let me know if you're not the right POC for that, but the move to GA has led to various significant decrease in workflow qualities that I hope we can address soon.

@osalpekar
Copy link
Member

@NicolasHug We're the POC's for that. Would love to set up some time and learn about your pain points with GHA CI jobs, and how the user experience can be better.

@NicolasHug
Copy link
Member

Thank you @osalpekar ,

@pmeier just reminded me that he's planning on sharing some of that feedback with the nova team soon. I'll make sure to sync with him so he can report our experience!

@NicolasHug
Copy link
Member

NicolasHug commented Feb 22, 2023

The 3.10 failure is unrelated. It's a CI/infra issue that needs to be fixed asap - I added it as the first entry in #7299. This PR is safe to merge.

@pmeier pmeier merged commit 49c6961 into pytorch:main Feb 22, 2023
@pmeier pmeier deleted the gha-ci branch February 22, 2023 10:34
@github-actions
Copy link

Hey @pmeier!

You merged this PR, but no labels were added. The list of valid labels is available at https://github.com/pytorch/vision/blob/main/.github/process_commit.py

NicolasHug pushed a commit to NicolasHug/vision that referenced this pull request Feb 24, 2023
facebook-github-bot pushed a commit that referenced this pull request Mar 29, 2023
Summary: Co-authored-by: vfdev <[email protected]>

Reviewed By: vmoens

Differential Revision: D44416565

fbshipit-source-id: 63c0da375d4bb143bd2cf78bbfd92878ddfa21cc
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants