Support deploying Kafka with KRaft instead of ZooKeeper for consensus building

[soenkeliebau]

In later versions Kafka supports running without ZooKeeper and provides its own KRaft consensus algorithm.

We should look into enabling the operator to deploy Kafka in this mode.

Refinement

• Make zookeeperConfigMapName optional and deprecate. Not required anymore once only 4.x is supported.

pub struct KafkaClusterConfig {
    [...]
    #[deprecated(since="...")]
    pub zookeeper_config_map_name: Option<String>,
}

• Add Controller Role (optional as broker)
• Sanity checks for kraft & zk relevant fields
• Provision log dirs with meta.properties
• New set of integration tests
• Documentation

Out of scope

Migration guide: https://docs.confluent.io/platform/current/installation/migrate-zk-kraft.html

• test manually with 3.9.0 and decide later if automation is feasible (or document manual steps)
• automated migration is out of scope
• Users need to migrate to KRaft (with 3.9.0) before upgrading to 4.0.0.

[sbernauer]

I just wanted to mention that according to https://www.confluent.io/blog/zookeeper-to-kraft-with-confluent-kubernetes/:

With Kafka 4.0 and Confluent Platform 8.0, Kafka will complete its journey to KRaft by removing the deprecated ZK mode,

So we need to tackle this at some point of time

[NickLarsenNZ]

I assume we want a CRD change to remove the Zookeeper ConfigMap reference.

There will be a CRD versioning discussion during the on-site about how to version CRDs (eg: how to choose from v1alpha2 vs v1beta1 vs v2alpha1).