[installer] Add support for Ingress traffic in cloud providers

[aledbf]

Bug description

The installer uses services type=ClusterIP. In particular for workspace clusters we need to use type=LoadBalancer, using standard ports 80 and 443. If we could also add support for additional annotations it would be ideal.

The goal here is to reduce the need for post-processing.

Steps to reproduce

Workspace affected

No response

Expected behavior

No response

Example repository

No response

Anything else?

No response

[corneliusludmann]

Thanks, @aledbf, for opening this issue.

Could you help me better understand what we actually need?

1. Does a workspace cluster always need the ingress type LoadBalancer instead of ClusterIP (even for self-hosted installations) or does it depend? The question is about whether we can simply change the ingress for the case InstallatinKind = "Workspace" or if we need a configuration option for that.
   We have so far refrained from making this freely configurable because otherwise, we would also have to fully support it. So it would be good to understand first, what we really need.
2. What support for additional annotations do you have in mind? @metcalfc once said that customers often need a specific set of annotations (e.g. for security policies) and that we maybe should support to configure a list of annotations in the config file. Again, it would be good to better understand the use cases here.

Thanks. 🙏

[aledbf]

What support for additional annotations do you have in mind?

In proxy, ws-proxy or ws-manager, depending the kind of installation, if you are in GCP for instance, you need to set
cloud.google.com/neg in the service.

[aledbf]

Does a workspace cluster always need the ingress type LoadBalancer instead of ClusterIP (even for self-hosted installations) or does it depend?

I think this is more related where are you running Gitpod than anything else. If you are running in the cloud (not important which one) you will use a service type LoadBalancer.

There is one exception, EKS/AWS, where you need to use Ingress (ALB) in order to avoid websocket disconnections

[aledbf]

We have so far refrained from making this freely configurable because otherwise, we would also have to fully support it. So it would be good to understand first, what we really need.

I share that idea until I need to do this post-processing, which is weird and not easy to maintain

yq e --inplace 'select(.kind == "Service" and .metadata.name == "ws-proxy").spec.type="LoadBalancer"' manifests.yaml
yq e --inplace 'select(.kind == "Service" and .metadata.name == "ws-proxy").spec.ports=[{"name": "http-proxy","port": 80,"protocol": "TCP","targetPort": 8080},{"name": "https-proxy","port": 443,"protocol": "TCP","targetPort": 9090},{"name": "ssh","port": 22,"protocol": "TCP","targetPort": 2200}]'

[yevon]

The [installer] already supports cert-manager, it should support ingress and annotations. I got it working after realizing that I needed redirecting to internal port 443 and setting backend protocol as https. Find attached my ingress for nginx, as this might help others:

`
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-gitpod
namespace: gitpod
annotations:
cert-manager.io/issuer: "letsencrypt-prod"
nginx.ingress.kubernetes.io/proxy-body-size: "6000000m"
nginx.org/client-max-body-size: "6000000m"
nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
nginx.ingress.kubernetes.io/permanent-redirect-code: "301"
spec:
tls:

• hosts:
  • "gitpod.domain.com"
  • "*.gitpod.domain.com"
  • "*.ws.gitpod.domain.com"
    secretName: https-certificates
    rules:
• host: "gitpod.domain.com"
  http:
  paths:
  • path: /
    pathType: Prefix
    backend:
    service:
    name: proxy
    port:
    number: 443
• host: "*.gitpod.domain.com"
  http:
  paths:
  • path: /
    pathType: Prefix
    backend:
    service:
    name: proxy
    port:
    number: 443
• host: "*.ws.gitpod.domain.com"
  http:
  paths:
  • path: /
    pathType: Prefix
    backend:
    service:
    name: proxy
    port:
    number: 443
    ingressClassName: nginx
    `

[aledbf]

it should support ingress and annotations.

My request is to support service type=LoadBalancer, not Kubernetes Ingress.

One of the reasons we don't support Ingress OOTB is that some implementations, like NGINX, require a reload every time you change and Ingress or a Secret, which disconnects all WebSocket sessions.

[yevon]

But it is not so common to make changes to the ingress once you have set it up, and when you do so you can assume that some things might stop working temporarily. Or by doing so something can stop working permanently? Anyway this could happen anyway when you update gitpod isn't it?