Prebuilds are stuck in pending state

[princerachit]

Bug description

Several prebuilds of my project are stuck in pending state.
On investigating I saw their state in database was queued.

I can see that there are 5K+ prebuilds in queued state.

select count(*) from d_b_prebuilt_workspace where creationTime > '2022-04-18 06:30:28.948681 UTC' and creationTime < '2022-04-19 06:30:28.948681 UTC' and state = 'queued';

5048

Steps to reproduce

Push several branches in your project's repo.
You will see all the prebuilds getting stuck in pending state.

For some branches e.g. main prebuilds might run and succeed.

Workspace affected

No response

Expected behavior

Prebuilds should run after few minutes in pending state.

Example repository

https://github.com/princerachit/pub

Anything else?

No response

[svenefftinge]

While the reported issue seem to have been caused by an db-sync problem, I see ~6k queued entries without the date filters. Some of them from 2019.

[princerachit]

I am closing this issue as the related incident has been resolved. @svenefftinge maybe we can check with @gitpod-io/engineering-webapp team about those entries and create a separate issue if required.

[svenefftinge]

ok done #9395

[geropl]

@svenefftinge Ok you were faster. Copying over to #9395 s description.