Implement airflow triggerer

[adwk67]

Description / Context

A running triggerer process allows deferrable operators to be used in DAGs so that worker management can be improved e.g. on scheduler re-starts and for providing HA: see here for more details.

Problem

Currently the airflow-operator does not implement an airflow triggerer process/role.

Proposal

Introduce a new optional role to go alongside the existing ones:

• webservers
• schedulers
• celeryExecutors / kubernetesExecutors
• triggerers

This new role will encapsulate a command that starts the triggerer.

The CRD would look something like this:

---
apiVersion: airflow.stackable.tech/v1alpha1
kind: AirflowCluster
metadata:
  name: airflow
spec:
  ...
  webservers:
    ...
  celeryExecutors:
    ...
  schedulers:
    ...
  triggerers:
    roleGroups:
      default:
        replicas: 1

Acceptance criteria

• add an airflow role triggerer
• create a test that uses a DAG/deferrable operator that keeps running over a re-deployment/scheduler re-start

[lfrancke]

Thanks. This is fine for me.

[adwk67]

Release note snippet

The Airflow triggerer component is now supported. This can be used with DAGs utilising deferrable operators to keep worker slots free and enhance HA.

[adwk67]

Completed by #679

[lfrancke]

Do we have docs for this?

[adwk67]

Docs (for dag-processors and triggerers) can be found here:

• https://docs.stackable.tech/home/nightly/airflow/getting_started/first_steps/#_airflow
• https://docs.stackable.tech/home/nightly/airflow/#_dag_processors
• https://docs.stackable.tech/home/nightly/airflow/#_triggerers