Skip to content

Clean up webhook events #12472

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Aug 31, 2022
Merged

Clean up webhook events #12472

merged 1 commit into from
Aug 31, 2022

Conversation

AlexTugarev
Copy link
Member

@AlexTugarev AlexTugarev commented Aug 29, 2022

by running a periodic garbage collector. This is a pattern we use for other resources.

fixes #12430

Description

The way GC works is admittedly tricky, as it's an interplay between the periodic GC task and the db-sync. Periodically this will mark a bunch of entries as deleted, and we rely on the db-sync to actually delete the entries.

Because it's proven unreliable to mark entries as delete in the thousands, leaving them in the db for ever after a failure, the settings to mark as deleted periodically are: 600 entries every 6 minutes, which is a rate of ~200K/day and should not produce a lot of DB CPU usage.

Related Issue(s)

Fixes #12430

How to test

Release Notes

NONE

Documentation

Werft options:

  • /werft with-preview

@AlexTugarev AlexTugarev requested a review from a team August 29, 2022 14:16
@github-actions github-actions bot added the team: webapp Issue belongs to the WebApp team label Aug 29, 2022
@werft-gitpod-dev-com
Copy link

started the job as gitpod-build-at-events-gc.1 because the annotations in the pull request description changed
(with .werft/ from main)

@AlexTugarev
Copy link
Member Author

back to draft. just tested the query and explain showed me a full table scan :-P

@AlexTugarev AlexTugarev marked this pull request as ready for review August 30, 2022 06:01
@AlexTugarev
Copy link
Member Author

I removed the problematic delete <> 0 predicate. Also adding more description on how this will work in the end.

@AlexTugarev
Copy link
Member Author

@geropl, maybe you want to have a look at this one?

@geropl
Copy link
Member

geropl commented Aug 30, 2022

@AlexTugarev Will review now.

@geropl geropl self-assigned this Aug 30, 2022
by running a periodic garbage collector. This is a pattern we use for other resources.

fixes #12430
@AlexTugarev
Copy link
Member Author

AlexTugarev commented Aug 30, 2022

/werft run

👍 started the job as gitpod-build-at-events-gc.5
(with .werft/ from main)

@AlexTugarev
Copy link
Member Author

build was failing for an unrelated issue with db-sync replication. restarted.

@AlexTugarev
Copy link
Member Author

AlexTugarev commented Aug 31, 2022

/werft run

👍 started the job as gitpod-build-at-events-gc.6
(with .werft/ from main)

@AlexTugarev AlexTugarev requested a review from geropl August 31, 2022 13:03
Copy link
Member

@geropl geropl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code LGTM, tested and saw the GC doing it's job. Thank you! 🎉

@roboquat roboquat merged commit a2fa9dc into main Aug 31, 2022
@roboquat roboquat deleted the at/events-gc branch August 31, 2022 13:08
@roboquat roboquat added deployed: webapp Meta team change is running in production deployed Change is completely running in production labels Aug 31, 2022
@AlexTugarev AlexTugarev mentioned this pull request Aug 31, 2022
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
deployed: webapp Meta team change is running in production deployed Change is completely running in production release-note-none size/M team: webapp Issue belongs to the WebApp team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Clean up recorded webhook events
3 participants