Skip to content

[supervisor] Properly terminate background tasks #2654

Closed
@csweichel

Description

@csweichel

When a Gitpod task starts a process in the background (e.g. sudo docker-up &) that process might not be subject to the same termination rules that other children of supervisor are.

When supervisor is asked to shut down (receives SIGINT/SIGTERM) it sends SIGINT to its terminals followed by a SIGKILL a few seconds later.

We should investigate what happens to background processes started by supervisor's task in this scenario.
Do they receive the same signal regimen, are they SIGKILL'ed or reparented to some supervisor process?

Activity

added this to the January 2021 milestone on Jan 4, 2021
self-assigned this
on Jan 8, 2021
csweichel

csweichel commented on Jan 13, 2021

@csweichel
ContributorAuthor

The motivation for this issue is running docker-up during prebuilds. Prebuilds finish when all their tasks finish, which means we can't just run sudo docker-up in a prebuild. Instead, we have a few options:

sudo docker-up &

  • 👍 comes built-in with the bash, it's nothing we have to implement
  • 👍 it's somewhat "intuitive", i.e. it's what more experienced users would do once they know that all tasks have to terminate
  • 👎 bash does not forward sigint/sigterm to background processes but kills them hard. This prevents orderly shutdown of the background task.

discover background tasks and gracefully terminate

We'd have the supervisor inspect the process hierarchy, detect background tasks and terminate them with the same shutdown procedure it uses for terminals (sigterm followed by sigkill).

gp daemonize

We'd introduce a gp daemonize (e.g. gp daemonize -- sudo docker-up) command which would fork the child and then exit, making sure the child is reparented to the PID1 process beforehand.

  • 👍 adds a neat runtime-level annotation to "background tasks"
  • 👎 it's unclear how this would actually work, i.e. if/how we can do that reparenting
  • 👎 it's something unusual, i.e. not something other environments/systems use

background tasks in .gitpod.yml

We'd introduce a special background task, or possibly a marker on the tasks whereby prebuilds would no longer wait for those tasks to complete.

  • 👍 it's a very explicit mechanism and clearly communicates how prebuild tasks behave
  • 👎 it's fairly intrusive and requires changes to quite a few bits within Gitpod (config, supervisor)
  • 👎 semantics aren't immediately clear to users. How would a background task differ from an init task? If we wanted to annotate existing tasks instead, how would we go about that? A task is defined by its command only.
added 7 commits that reference this issue on Jan 14, 2021
28ad4ce
408ae76
weitzman

weitzman commented on Feb 14, 2021

@weitzman

So is it now possible to use dockerd in prebuild?

6 remaining items

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Relationships

None yet

    Development

    No branches or pull requests

      Participants

      @weitzman@shaal@akosyakov@csweichel

      Issue actions

        [supervisor] Properly terminate background tasks · Issue #2654 · gitpod-io/gitpod