Skip to content

Increase CI timeout #8173

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Feb 4, 2025
Merged

Increase CI timeout #8173

merged 1 commit into from
Feb 4, 2025

Conversation

metascroy
Copy link
Contributor

Since #7922, 2 CI jobs consistently hit timeout.

@pytorch-bot PyTorch Bot
Copy link

pytorch-bot bot commented Feb 4, 2025 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/8173

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit dd142f8 with merge base 9441003 (image):
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Feb 4, 2025
@metascroy metascroy added the module: ci Issues related to continuous integration label Feb 4, 2025
@metascroy metascroy requested a review from swolchok February 4, 2025 03:46
@metascroy
Copy link
Contributor Author

@pytorchbot label "topic: not user facing"

@swolchok
Copy link
Contributor

swolchok commented Feb 4, 2025

Are people really going to wait this long? I already had no patience with the jobs taking an hour.

I looked into it briefly and I dont know why they would be taking longer; the pinned PyTorch commit is prebuilt at docker container generation time and I’m fairly sure it is in fact a release build.

@swolchok
Copy link
Contributor

swolchok commented Feb 4, 2025

Something is definitely wrong. In the raw logs for the first test-llava-runner-linux timeout on main, there are almost 40 minutes of "gaps" in the logs with no timestamps. Specifically:

  • 14 minute "gap" in logs, jump from 2025-01-31T23:11:50.6702881Z to 2025-01-31T23:25:21.0383351Z, during export.
  • 25 minute "gap in logs from 2025-01-31T23:25:21.2243143Z to 2025-01-31T23:42:40.6914293Z , and the second message is just a job timeout.

@swolchok swolchok mentioned this pull request Feb 4, 2025
Closed
@swolchok
Copy link
Contributor

swolchok commented Feb 4, 2025

tracking issue: #8180

@metascroy
Copy link
Contributor Author

Are people really going to wait this long? I already had no patience with the jobs taking an hour.

I looked into it briefly and I dont know why they would be taking longer; the pinned PyTorch commit is prebuilt at docker container generation time and I’m fairly sure it is in fact a release build.

It's a good question, but currently this is one of the issues blocking viable/strict from updating and I thought having a short-term "fix" as the investigation continues would be better than nothing.

There are several ways we could fix:

swolchok
swolchok approved these changes Feb 4, 2025
View reviewed changes
Copy link
Contributor

@swolchok swolchok left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

unblocking seems good

@metascroy metascroy merged commit 25707ea into pytorch:main Feb 4, 2025
45 of 48 checks passed
@swolchok swolchok mentioned this pull request Feb 10, 2025
Merged
swolchok added a commit that referenced this pull request Feb 10, 2025
@swolchok
Lower timeout for llava-runner and eval_llama-mmlu jobs again
878f14c 
#8173 raised these timeouts. Now that #8248 has landed to fix #8180, we should be able to lower them again. (I'm sending this early so I don't forget; double-check llava-runner running time)

ghstack-source-id: cb4c169
ghstack-comment-id: 2648474106
Pull Request resolved: #8339
This was referenced Feb 11, 2025
Open
Open
This was referenced Feb 24, 2025
Open
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@swolchok swolchok swolchok approved these changes

Assignees
No one assigned
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. module: ci Issues related to continuous integration topic: not user facing
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@metascroy @swolchok @facebook-github-bot