[self-hosted] Add officially support service type ClusterIP for proxy

[corneliusludmann]

With #10537, we support setting the service type of the proxy via an experimental config option in the installer. We see, that we need to support this officially (internal discussion).

For official support, we need:

• Proposed documentation #10983
• Move loadbalancer-specific annotations to loadbalancer-only #11025
• Move installer config value out of experimental to the main section. #11008
• Add a config option in KOTS. #11009
• Write a docs page that explains the usage. #11011
• Make sure it will be tested (could be: unit tests, part of an explicit dimension in the automated self-hosted tests, part of a reference architecture that will be tested by the automated tests, etc). #11033

[lucasvaltl]

Replaces #9981

[szab100]

@corneliusludmann @lucasvaltl When implementing this, can we move adding these two hard-coded annotations to inside the IF(type == LB) {..} stmt above or remove the hard-coded entries completely and just document in which cases should these be added to ProxyConfig.ServiceAnnotations?

Generally, these should only be used when 'proxy' is a load-balancer. When it's NodePort or ClusterIP, the 'external-dns' annotation on 'proxy' can interfere with other ingress / load-balancer resources that can also be using this annotation with same host values. This creates a race condition in the external dns system, causing the LB to be unreachable.

[mrsimonemms]

@corneliusludmann should this also remove the ClusterIP from the experimental section?

[mrsimonemms]

In "blocked" until #11011 is merged when July release is published

[mrzarquon]

@mrsimonemms if this setting moves from experimental to main, we should still honor the setting if it's in experimental and flag it with a deprecation warning. otherwise people who had clusterip working in 2022.6 will have it stop working in 2022.7 if they don't update the config (and since it's not easy to download / update the config in the replicated UI, there's no way for them to know)

[mrsimonemms]

Yes, we still honour the experimental setting and will do until December release #11124

As far as I know, there's no way of flagging a deprecation in KOTS dashboard

[mrzarquon]

gitpod-installer parses config, outputs to a assessment.log, troubleshoot.sh analyzer checks for "has_deprecated_experimental_feature" and reports it?

[mrsimonemms]

gitpod-installer parses config, outputs to a assessment.log, troubleshoot.sh analyzer checks for "has_deprecated_experimental_feature" and reports it?

Do you want to raise this as a separate ticket?