[Proposal] Use queues to distribute Actions tasks to runners (but it could be tricky)

[wolfogre]

Please note that this is not a real proposal.

Instead of discussing "how to use a queue to distribute tasks," I'm going to talk about the difficulties that arise when using a queue to distribute tasks. Despite its potential benefits, there are certain drawbacks that make it not the most effective way to distribute tasks.

This is just my opinion, and I have no objection to others supporting and implementing it. However, I hope them to know some details before doing so.

Background

Tasks are always reminiscent of queues, actually, we did try to use a queue to distribute tasks, but in the end, we did not use this approach. Instead, we only query the database and find a new task when a runner requests one.

However, frequently querying the database can be expensive. It's natural to wonder if we can store tasks in a queue. Querying the queue frequently is much cheaper.

Replace proposal #24544 if this one is feasible.

Before we go any further, let's keep the queue model provided by Gitea in mind. Or we have to rewrite a new implementation.

gitea/modules/queue/base.go

Lines 13 to 20 in b3af748

	type baseQueue interface {
	PushItem(ctx context.Context, data []byte) error
	PopItem(ctx context.Context) ([]byte, error)
	HasItem(ctx context.Context, data []byte) (bool, error)
	Len(ctx context.Context) (int, error)
	Close() error
	RemoveAll(ctx context.Context) error
	}

So

• We cannot preview the head item without poping it.
• We cannot remove an item which isn't the head item.

Considered solutions

1. Store all tasks in one queue

It's totally unworkable.

Because tasks could have different runs-on, consider the following queue:

Queue
task 1, runs on ubuntu
task 2, runs on windows
task 3, runs on ubuntu

When a runner labeled windows tries to get a new task, they may receive the wrong task (task 1, runs on ubuntu) or have to wait until task 1 is picked by another runner.

2. Store tasks in multiple queues, each queue is for a specific runner

Queue for runner X	Queue for runner Y	Queue for runner Z
task 1, runs on ubuntu	task 3, runs on ubuntu	task 2, runs on windows
task 4, runs on ubuntu	task 5, runs on ubuntu
task 6, runs on ubuntu

If we want to do this, we have to assign each task to a specific runner before the runner requests a new task. But that can cause a lot of problems:

• When the queue for runner X gets empty, but runner Y gets blocked by task 3 (it's a heavy task), runner X cannot help to run task 5.
• We have to restructure the queues if a runner got down or a new runner joined.
• We have to restructure the queues if a runner updated its labels, it’s possible after [Proposal] Support changing labels of Actions runner without re-registration #24540.

3. Store tasks in multiple queues, each queue is for a specific label

Queue for ubuntu	Queue for windows
task 1, runs on ubuntu	task 3, runs on windows
task 4, runs on ubuntu	task 5, runs on windows
task 6, runs on ubuntu

I would say it will work fine, but not ideally, because it could be tricky to maintain the set of labels, whether we get the labels from workflow files or registered runners.

After a period of time, the queues could be like:

Queue for ubuntu	Queue for windows	Queue for ubuntuuuuu it's caused by a typo which has already been fixed	Queue for test_arm_linux it's caused by a test runner which has already been removed	And hundreds of other useless queues
task 1, runs on ubuntu	task 3, runs on windows
task 4, runs on ubuntu	task 5, runs on windows
task 6, runs on ubuntu

Perhaps the issue could be resolved by periodically cleaning out unnecessary queues. However, how can we distinguish whether a queue is actually unnecessary or not? Maybe

• The queue has been empty for a long time.
  • It is not unusual for queues to be empty for long periods of time, even when they are in use.
• The label of the queue doesn’t exist in any workflows files.
  • Scan all workflow files in every commit? You know that we can trigger a workflow from any commit at any time.
• The label of the queue doesn’t exist in any runners.
  • What if a runner has been added/removed/updated?

More questions

• How to cancel a task that has already been enqueued?
• How to clean tasks that have not been picked up by a runner for a long time?