[supervisor] Properly terminate background tasks

[csweichel]

When a Gitpod task starts a process in the background (e.g. sudo docker-up &) that process might not be subject to the same termination rules that other children of supervisor are.

When supervisor is asked to shut down (receives SIGINT/SIGTERM) it sends SIGINT to its terminals followed by a SIGKILL a few seconds later.

We should investigate what happens to background processes started by supervisor's task in this scenario.
Do they receive the same signal regimen, are they SIGKILL'ed or reparented to some supervisor process?

[csweichel]

The motivation for this issue is running docker-up during prebuilds. Prebuilds finish when all their tasks finish, which means we can't just run sudo docker-up in a prebuild. Instead, we have a few options:

sudo docker-up &

• 👍 comes built-in with the bash, it's nothing we have to implement
• 👍 it's somewhat "intuitive", i.e. it's what more experienced users would do once they know that all tasks have to terminate
• 👎 bash does not forward sigint/sigterm to background processes but kills them hard. This prevents orderly shutdown of the background task.

discover background tasks and gracefully terminate

We'd have the supervisor inspect the process hierarchy, detect background tasks and terminate them with the same shutdown procedure it uses for terminals (sigterm followed by sigkill).

gp daemonize

We'd introduce a gp daemonize (e.g. gp daemonize -- sudo docker-up) command which would fork the child and then exit, making sure the child is reparented to the PID1 process beforehand.

• 👍 adds a neat runtime-level annotation to "background tasks"
• 👎 it's unclear how this would actually work, i.e. if/how we can do that reparenting
• 👎 it's something unusual, i.e. not something other environments/systems use

background tasks in .gitpod.yml

We'd introduce a special background task, or possibly a marker on the tasks whereby prebuilds would no longer wait for those tasks to complete.

• 👍 it's a very explicit mechanism and clearly communicates how prebuild tasks behave
• 👎 it's fairly intrusive and requires changes to quite a few bits within Gitpod (config, supervisor)
• 👎 semantics aren't immediately clear to users. How would a background task differ from an init task? If we wanted to annotate existing tasks instead, how would we go about that? A task is defined by its command only.

[weitzman]

So is it now possible to use dockerd in prebuild?