Skip to content

Record webhook events #11356

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jul 15, 2022
Merged

Record webhook events #11356

merged 1 commit into from
Jul 15, 2022

Conversation

AlexTugarev
Copy link
Member

@AlexTugarev AlexTugarev commented Jul 13, 2022

Description

This PR adds recording of webhook events received for GH/GHE/GL/BB/BBS.

For each push event received for any configured repositories, a record will be stored in the DB containing the raw payload necessary to trigger a prebuild at any time. To start with a 🛹, the actual trigger remains untouched here, only the status is updated up the the point of a successful prebuild trigger (or any error case.) The status updates will be already beneficial in the follow up PR to render updates on the dashboard.

The proposed DB table is quite simple.

mysql> describe d_b_webhook_event;
+------------------+--------------+------+-----+----------------------+--------------------------------+
| Field            | Type         | Null | Key | Default              | Extra                          |
+------------------+--------------+------+-----+----------------------+--------------------------------+
| id               | char(36)     | NO   | PRI | NULL                 |                                |
| creationTime     | varchar(255) | NO   | MUL | NULL                 |                                |
| type             | char(60)     | NO   |     | NULL                 |                                |
| authorizedUserId | char(36)     | YES  |     | NULL                 |                                |
| status           | char(60)     | NO   |     | NULL                 |                                |
| message          | text         | YES  |     | NULL                 |                                |
| rawEvent         | text         | NO   |     | NULL                 |                                |
| cloneUrl         | char(255)    | YES  | MUL | NULL                 |                                |
| branch           | char(255)    | YES  |     | NULL                 |                                |
| commit           | varchar(255) | YES  |     | NULL                 |                                |
| projectId        | char(36)     | YES  |     | NULL                 |                                |
| prebuildStatus   | char(60)     | YES  |     | NULL                 |                                |
| prebuildId       | char(36)     | YES  |     | NULL                 |                                |
| deleted          | tinyint(4)   | NO   |     | 0                    |                                |
| _lastModified    | timestamp(6) | NO   | MUL | CURRENT_TIMESTAMP(6) | on update CURRENT_TIMESTAMP(6) |
+------------------+--------------+------+-----+----------------------+--------------------------------+
15 rows in set (0.02 sec)

Related Issue(s)

Related #10764

How to test

Push a commit to a repository configured as a project with this preview environment and verify you see webhook events stored in the DB.

mysql> select type, status, prebuildStatus from d_b_webhook_event;
+------+-----------+--------------------+
| type | status    | prebuildStatus     |
+------+-----------+--------------------+
| push | processed | prebuild_triggered |
+------+-----------+--------------------+
1 row in set (0.00 sec)

Release Notes

NONE

Documentation

Werft options:

  • /werft with-preview
  • /werft with-clean-slate-deployment

@AlexTugarev
Copy link
Member Author

AlexTugarev commented Jul 13, 2022

/werft run

👍 started the job as gitpod-build-at-webhook-events.7
(with .werft/ from main)

@AlexTugarev AlexTugarev marked this pull request as ready for review July 13, 2022 16:04
@AlexTugarev AlexTugarev requested a review from a team July 13, 2022 16:04
@github-actions github-actions bot added the team: webapp Issue belongs to the WebApp team label Jul 13, 2022
@AlexTugarev
Copy link
Member Author

@geropl, please have a look on the DB table.

@geropl geropl self-assigned this Jul 14, 2022
const config = await this.prebuildManager.fetchConfig({ span }, user, context);
if (!this.prebuildManager.shouldPrebuild(config)) {
console.log("Bitbucket push event: No config. No prebuild.");
await this.webhookEvents.updateEvent(event.id, { status: "ignored_unconfigured" });
Copy link
Member

@geropl geropl Jul 14, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm. As the table is about WebhookEvents now this prebuild-specific status does not make sense now: Imagine a webhook triggered both, a preview and a prebuild, what status would it be in?

What about keeping the status strictly with the WebHook:

"received"
  | "dismissed_unauthorized"
  | "ignored_incomplete"
  | "failed" // generic error during _webhook_ processing
  | "processed" // or similar

Still, we need a place to document the decisions. E.g. whether or not a prebuild got triggered is conveyed by the prebuildId, but still we need to store the "why", ideally in an indexable + a human readable format (think status + message). So far this has been done inside the PrebuiltWorkspace entity, but I really really would like to get away from that. 🤔

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

prebuild-specific status does not make sense now: Imagine a webhook triggered both, a preview and a prebuild, what status would it be in?

Good question. I think instead of generalizing to a minimal set of commons, I'd rather split the status by concerns. In fact, we need explicit states for the frontend to render. If we coerce different concerns into generic states, I fear it will be difficult to render meaningful states.

What about adding explicit prebuildStatus? We can tag specific statuses.

Also, is there something to read more on the new preview feature? I find it hard to reason about, as I've no details so far.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The preview feature will. basically start a workspace on those events in a similar context as prebuilds. The main difference is that it keeps running longer (long timeout + lifecycle bound to commit on tip of branch) and exposes a port publicly which is sent back to GH/GL as a check.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe the webhook table should not contain any prebuild information at all. In fact I think even the reference should be the other way around. I.e. a prebuild that was triggered based on a webhook should reference it.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe the webhook table should not contain any prebuild information at all.

It's certainly a trade off, but I feel that it's worth it. The goal here is to untangle the question "what happened to this prebuild event and why" from the PrebuiltWorkspace state machine (which feels broken atm).

We could certainly add another table, e.g. WebhookEvent -> PrebuildEvent -> PrebuiltWorkspace, but as we'd join that on primary key of the WebHook event and to avoid the additional complexity here, I think it's ok (for now) to have that inside the WebhookEvent table. 🧘

@svenefftinge
Copy link
Member

Could we also already extract the pushing user and match it with our user? We plan to build a metric into the product that shows team activation and will be based on the difference of commits authored in total (the webhook events) vs. which of those were done from Gitpod. So would be great to have a user shape in this table that optionally has a userID if that user exists in d_b_user.

@geropl
Copy link
Member

geropl commented Jul 14, 2022

Could we also already extract the pushing user and match it with our user? We plan to build a metric into the product that shows team activation and will be based on the difference of commits authored in total (the webhook events) vs. which of those were done from Gitpod. So would be great to have a user shape in this table that optionally has a userID if that user exists in d_b_user.

I think that is the intend, but this PR serves as a first step towards that goal. 👣 👍

@svenefftinge
Copy link
Member

svenefftinge commented Jul 14, 2022

Also, extracting branch name and commit seem reasonable? 😇

@AlexTugarev AlexTugarev force-pushed the at/webhook-events branch 2 times, most recently from 9265737 to f1c3405 Compare July 14, 2022 12:54
@AlexTugarev
Copy link
Member Author

AlexTugarev commented Jul 14, 2022

/werft run

👍 started the job as gitpod-build-at-webhook-events.10
(with .werft/ from main)

Copy link
Member

@geropl geropl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code LGTM, tested and works! 🚀
Super exited to have this, thank you @AlexTugarev ! 😍

@roboquat roboquat merged commit 6c395b5 into main Jul 15, 2022
@roboquat roboquat deleted the at/webhook-events branch July 15, 2022 08:58
@roboquat roboquat added deployed: webapp Meta team change is running in production deployed Change is completely running in production labels Jul 19, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
deployed: webapp Meta team change is running in production deployed Change is completely running in production release-note-none size/XL team: webapp Issue belongs to the WebApp team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Record a "prebuild trigger" for every webhook event
4 participants