Skip to content

[Proposal] Reduce unnecessary DB queries for Actions tasks #24544

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wolfogre opened this issue May 5, 2023 · 0 comments · Fixed by #25199
Closed

[Proposal] Reduce unnecessary DB queries for Actions tasks #24544

wolfogre opened this issue May 5, 2023 · 0 comments · Fixed by #25199
Assignees
Labels
topic/gitea-actions related to the actions of Gitea type/proposal The new feature has not been accepted yet but needs to be discussed first.

Comments

@wolfogre
Copy link
Member

wolfogre commented May 5, 2023

This looks familiar with #24543, but it's quite different.

Background

Gitea will start a transaction to find and assign a task to the runner when it requests a new one. However, we know that there may not be a task available most of the time, so Gitea has to roll back the transaction and respond with "no task yet, try again later."

Starting a transaction is an expensive operation. However, we don't have to do it every time.

Solution

Record a version number for table of Actions jobs

PS: You might wonder why it's not "table of Actions tasks". Never mind, just a few implementation details.

Increase the version number once the data table changes. The number can be stored in DB and cached in memory.

When Gitea receives a request for a task, it compares the version number in the request with the current version number. If they are equal, it responds with "no task available yet".

Otherwise, start a transaction to query and assign a task.

Gitea responds with the current version number regardless of whether there is a task available.

Runners fetch tasks with the version number

  1. Fetch tasks with 0 version number when it’s the first time.
  2. Fetch tasks with the responded version number when it didn't get a task last time.
  3. Fetch tasks with 0 version number when it got a task last time.

Here's a simulation:

Gitea   Runner
  ← 0  
13 != 0, query db  → 13, no task  
  ← 13  
13 == 13, skip query  → 13, no task  
  ← 13  
13 != 14, query db  → 14, a task  
  ← 0  
14 != 0, query db  → 14, no task  
  ← 14  
14 == 14, skip query  → 14, no task  

Advantages

  • Easy to implement. No need to modify existing mechanisms.
  • Compatible with other improvement measures.
@wolfogre wolfogre added type/proposal The new feature has not been accepted yet but needs to be discussed first. type/feature Completely new functionality. Can only be merged if feature freeze is not active. topic/gitea-actions related to the actions of Gitea labels May 5, 2023
@wolfogre wolfogre changed the title [Proposal] Reduce unnecessary DB queries for Actions tasks. [Proposal] Reduce unnecessary DB queries for Actions tasks May 9, 2023
@wolfogre wolfogre removed the type/feature Completely new functionality. Can only be merged if feature freeze is not active. label May 9, 2023
@sillyguodong sillyguodong self-assigned this May 24, 2023
lunny pushed a commit that referenced this issue Jul 24, 2023
Close #24544

Changes:

- Create `action_tasks_version` table to store the latest version of
each scope (global, org and repo).
- When a job with the status of `waiting` is created, the tasks version
of the scopes it belongs to will increase.
- When the status of a job already in the database is updated to
`waiting`, the tasks version of the scopes it belongs to will increase.
- On Gitea side, in `FeatchTask()`, will try to query the
`action_tasks_version` record of the scope of the runner that call
`FetchTask()`. If the record does not exist, will insert a row. Then,
Gitea will compare the version passed from runner to Gitea with the
version in database, if inconsistent, try pick task. Gitea always
returns the latest version from database to the runner.

Related:

- Protocol: https://gitea.com/gitea/actions-proto-def/pulls/10
- Runner: https://gitea.com/gitea/act_runner/pulls/219
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 8, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
topic/gitea-actions related to the actions of Gitea type/proposal The new feature has not been accepted yet but needs to be discussed first.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants