Gitpod on Self-Hosted Rancher without Cloud Provider

[lucky4ever2]

I have a self hosted rancher without a cloud provider. Therefore the proxy cannot create a load balancer service.
I changed the serviceType to "NodePort" and created an ingress behind it at Port 80. But unfortunately I get the default nginx page displayed: "Welcome to nginx!" (https://gitpod.domain.de)

My Helm Chart Config:
hostname=gitpod.domain.de
certificatesSecret.secretName=gitpod-ssl-secret
ingressMode=pathAndHost
components.proxy.serviceType=NodePort
minio.persistence.size=50Gi
components.wsDaemon.containerRuntime.nodeRoots[0]=/var/lib
components.wsDaemon.containerRuntime.nodeRoots[1]=/run/containerd/io.containerd.runtime.v1.linux/moby
defaults.dnsPolicy=ClusterFirstWithHostNet

And I don't know if that is related, but the ws-manager-bridge shows this error:
Error in fetching sampling strategy: Error: connect ECONNREFUSED 0.0.0.0:5778

[csweichel]

And I don't know if that is related, but the ws-manager-bridge shows this error:
Error in fetching sampling strategy: Error: connect ECONNREFUSED 0.0.0.0:5778

That one's unrelated. It's Jaeger/distributed tracing that's complaining. Entirely optional.

It would seem that we're breaking some expectation of the proxy here. Maybe @geropl can say more about this?

[geropl]

HI @lucky4ever2 ,

I'm no sure what the problem is here, but I also do not understand your setup entirely: Could you share the config of your ingress setup?
In general it should be enough to switch the service type to NodePort (targetPort: 443, because nginx usually does ssl termination) and forward all traffic to hostIp:nodePort.

Also, pathAndHost might be the problem here, and is likely to be removed in the future: #2683

Would it be an option to create and use wildcard certificates as described here and switch to ingressMode=hosts?

[lucky4ever2]

Hi @csweichel and @geropl,
thank you for your feedback.

The ingress behind the proxy is an L7 ingress. It does the SSL termination. When I use ingressMode = hosts the browser says redirect error because the proxy is doing the SSL termination and the ingress is doing the SSL termination. I've already tried the Ingress: kubernetes.io/ssl-redirect=false. But the same error occurs.
My ingress configuration: (censored)

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    field.cattle.io/creatorId: user-rxzkz
    field.cattle.io/ingressState: '{"Z2l0cGdHp3ZXJrcGxhZQ==":"p-l526g:gitpod-ssl-secret","Z2l0cG9kLWluZ3Jlc3ya3W4uZGUvLzgw":""}'
    field.cattle.io/publicEndpoints: '[{"addresses":["XXX.XXX.XXX.XXX"],"port":443,"protocol":"HTTPS","serviceName":"gitpod:proxy","ingressName":"gitpod:gitpod-ingress","hostname":"gitpod.domain.de","allNodes":true}]'
  creationTimestamp: "2021-01-20T08:41:17Z"
  generation: 1
  labels:
    cattle.io/creator: norman
  managedFields:
  - apiVersion: networking.k8s.io/v1beta1
    fieldsType: FieldsV1
    fieldsV1:
      f:status:
        f:loadBalancer:
          f:ingress: {}
    manager: nginx-ingress-controller
    operation: Update
    time: "2021-01-20T08:41:52Z"
  - apiVersion: extensions/v1beta1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:field.cattle.io/creatorId: {}
          f:field.cattle.io/ingressState: {}
          f:field.cattle.io/publicEndpoints: {}
        f:labels:
          .: {}
          f:cattle.io/creator: {}
      f:spec:
        f:rules: {}
        f:tls: {}
    manager: rancher
    operation: Update
    time: "2021-01-20T08:41:52Z"
  name: gitpod-ingress
  namespace: gitpod
  resourceVersion: "29845404"
  selfLink: /apis/extensions/v1beta1/namespaces/gitpod/ingresses/gitpod-ingress
  uid: 0d20ab87-ee7e-4b211433219565
spec:
  rules:
  - host: gitpod.domain.de
    http:
      paths:
      - backend:
          serviceName: proxy
          servicePort: 80
        pathType: ImplementationSpecific
  tls:
  - hosts:
    - gitpod.domain.de
    secretName: gitpod-ssl-secret
status:
  loadBalancer:
    ingress:
    - ip: XXX.XXX.XXX.XXX
    - ip: XXX.XXX.XXX.XXX
    - ip: XXX.XXX.XXX.XXX

[geropl]

When I use ingressMode = hosts the browser says redirect error because the proxy is doing the SSL termination and the ingress is doing the SSL termination

Ok, I feared this. Currently one cannot simply switch HTTP/HTTPS on or off with one switch because it is mingled with/hidden behind the ingressMode.
Also, I think the reason this does not work is a bug in pathAndHost which we will deprecate soon.

To verify, could you check the content of /etc/nginx/vhost.server.conf (or find that file in the proxy-config-nginx ConfigMap)?

There should be a vhost with listen 80 and server_name {{ $domain }} www.{{ $domain }} _;
Also, did you check if the ingress forwards host-headers correctly? Looking at

gitpod/chart/config/proxy/vhost.server.conf

Lines 36 to 46 in ef1d791

	server {
	listen {{ $listen }} default_server;
	{{- if eq .Values.ingressMode "hosts" }}
	server_name {{ $domain }} www.{{ $domain }} _;
	{{- else }}
	{{- if $useHttps }}
	listen 80;
	{{- end }}
	include lib.workspace-locations-path.conf;
	{{- end }}

we might be missing a default_server in the pathAndHost case.

[lucky4ever2]

While I was comparing your vhost.server.conf, I noticed that the pathAndHost is missing server_name!
After i add it, gitpod works!

server {
    listen 443 ssl default_server;
    listen 80;
    **server_name gitpod.domain.de**;
    include lib.workspace-locations-path.conf;
    include lib.ssl.conf;
    include lib.https_redirect.conf;

{{- if $useHttps }}
    {{- if eq .Values.ingressMode "pathAndHost" }}
    listen 80;
    **server_name {{ $domain }} www.{{ $domain }} _;** <--- is the server_name missing here?
    {{- end }}
    include lib.ssl.conf;
    include lib.https_redirect.conf;
{{- end }}

But I don't understand why this mode should be switched off. In other Helm charts, there is always the option of using an ingress in the rancher-cluster without a load balancer (without a cloud provider). For example at Gitlab.

[geropl]

But I don't understand why this mode should be switched off. In other Helm charts, there is always the option of using an ingress in the rancher-cluster without a load balancer (without a cloud provider). For example at Gitlab.

ingressMode=pathAndHost is more about how individual workspaces are served, and was introduced as compatibility measure. Depending on which version of Gitpod Self-Hosted you use, it is already broken.

You can achieve the same result with ingressMode=hosts and type: NodePort without an Ingress.

(We are thinking about moving to Ingress for SSL termination etc. in the future, but that's not planned nor implemented yet).

[lucky4ever2]

Thank you for your help.
I now have ingressMode=hosts and an external nginx l4 loadbalancer that routes the 443 to the NodePort. It works

[stefanfritsch]

If anyone else wants SSL termination at the Ingress level, I've had some success by modifying the following line in chart/config/proxy/vhost.server.conf:

{{- $useHttps := eq (include "gitpod.scheme" $this) "https" -}}

e.g. change it to:

{{- $useHttps := .Values.proxyHttps -}}

you can then set forceHTTPS: true and proxyHTTPS: false in values.yaml and just create an Ingress with SSL-termination that points everything at the proxy.

By "some success" I mean that I've only tried it with the 0.6.0-beta and actually starting workspaces never worked due to #2956. But the proxy setup seemed to work fine.