Support own CA certificate / self-signed certificated for self-hosted installations

[corneliusludmann]

• Test what works and was not
• Write docs on how to use your own CA certificate
• Maybe add support to the installer if specific settings are needed
• Add support for CA in KOTS / Replicated installer

See also:

• [Self-Hosted] Support for own CA certificates #2615
• Support custom CA certificates in Helm #2984
• (internal)

[corneliusludmann]

Comment from @mrsimonemms (internal):

one thing that would make startup so much faster is if we didn't require a cert (or would allow a self-signed one). I did a little bit of exploration around this - we need to be able to turn off the HSTS headers emitted by proxy. That alone would allow us to run with self-signed ones

[csweichel]

If we could implement this without introducing too much variance (e.g. a special "custom CA cert" mode), that would be awesome! Those categorically variant modes don't age well (think "stage" as "prod" and "dev") :)

[mrsimonemms]

Agreed. My understanding is that we can get this working purely by allowing the HSTS headers in proxy to be configurable at deploy-time

[SirLemyDanger]

I saw (blocking) errors related to this:

1. registry facade (needs to connect to our gitlab container registry, which has a TLS cert signed by the company CA)
2. server (oauth towards gitlab, which has a TLS cert signed by the company CA)
3. ??? when opening a workspace

[VouDoo]

Hello 👋

I am facing the same issue that @SirLemyDanger explained.
In my environment, I have the following services:

• Self-hosted Gitpod
• Self-hosted GitLab server integrated with Gitpod
• Harbor used as image registry by Gitpod

All of those services use Company custom certificates (i.e. not trusted by a known public issuer).
When I try to create a workspace, I get this error (which does not say much):

After investigation, I was able to find error entries in the log of the registry-facade pod:

http: TLS handshake error from XXX.XXX.XXX.XXX:XXXXX: remote error: tls: bad certificate

Basically, Gitpod cannot trust the certificate from Harbor.

Is there a way to import the custom CA certificates to resolve that?

Thank you

[Pothulapati]

Hey @VouDoo ,

Can you tell us more about the environment in which the cluster is running? With self-signed, We have some known issues in GKE, as GKE containerd does not support custom certificate. 🤔

[VouDoo]

Hello @Pothulapati,

I'm running a self-hosted instance of Gitpod in a K3s cluster.
There is no issue with the installation/use of a custom (not self-signed) certificate on Gitpod.
The problem comes from the communication from Gitpod to the other services (GitLab, Harbor, etc.).
The Gitpod pods (e.g registry-facade) cannot trust any custom certificates, it has to be certificates issued by "official" CAs (retrieved by the ca-certificates package in Ubuntu containers).
Basically, I am looking for a solution to include a custom CA certificate inside every Gitpod containers' cacerts to trust the other services' custom certificates.

Thank you :)

[adrienthebo]

Hi @VouDoo,

The gitpod-installer exposes custom CA certificates by copying the system CA bundle and our internally generated CA certificate (for Gitpod's self-signed cert configuration option) into a Kubernetes volume, and mounts that volume into other containers to support our custom CA. Take a look at how that operates; you may be able to add your custom CA certificate for your other components to that bundle. If that doesn't work then we can break this discussion out into a new issue. Thanks!

[GitpodFrontApp]

Hey Maxence, Can you tell us more about the environment in which the cluster is running? With self-signed, We have some known issues in GKE, as GKE `containerd` does not support custom certificate. — Team Self-Hosted Community Captain Gitpod | @gitpod This is your Request ID: cnv_iucz34a How would you rate our help? {{survey}}

…

--- original message --- On June 17, 2022, 7:55 AM GMT+5:30 ***@***.*** wrote: Hello 👋 I am facing the same issue that @SirLemyDanger explained. In my environment, I have the following services: Self-hosted Gitpod Self-hosted GitLab server integrated with Gitpod Harbor used as image registry by Gitpod All of those services use Company custom certificates (i.e. not trusted by a known public issuer). When I try to create a workspace, I get this error (which does not say much): After investigation, I was able to find error entries in the log of the registry-facade pod: http: TLS handshake error from XXX.XXX.XXX.XXX:XXXXX: remote error: tls: bad certificate Basically, Gitpod cannot trust the certificate from Harbor. Is there a way to import the custom CA certificates to resolve that? Thank you — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

-- You received this message because you are subscribed to the Google Groups "gitpod-frontapp" group. To unsubscribe from this group and stop receiving emails from it, send an email to ***@***.*** To view this discussion on the web visit https://groups.google.com/a/gitpod.io/d/msgid/gitpod-frontapp/gitpod-io/gitpod/issues/8559/1158415292%40github.com. --- end of original message ---