Improve Error message for cert failures when launching preview envs.

[meysholdt]

Is your feature request related to a problem? Please describe

Currently, an error message looks as follows

Error: kubectl --kubeconfig /workspace/gitpod/kubeconfigs/core-dev wait --for=condition=Ready --timeout=600s -n certs certificate harvester-jenting-10334 exit with non-zero status code. STDERR: error: timed out waiting for the condition on certificates/harvester-jenting-10334

That is not helpful because it does not explain why no cert could be obtained.

Describe the behaviour you'd like

The error message should instead be the output of the comment cmctl status certificate <certname>.
This command summarises all Custom Resources that are involved in obtaining the certs.
This includes error events and status.

The error message should be clearly readable both in Werft and in HoneyComb.

Describe alternatives you've considered

We could debug into each job, understand the error better and fix the error - but I think it's more sustainable if we start by improving the error message first because it will give us more data on the problem and relieves us from the need to denug every job individually.

[liam-j-bennett]

I can add extra error output, but currently we don't seem to execute the code we have. (Please excuse, if I'm misunderstanding TS)

Currently, the code that provides this error is:

function waitForCertificateReadiness(werft: Werft, certName: string, slice: string) {
    const timeout = "600s"
    werft.log(slice, "Waiting for certificate readiness")
    const rc = exec(`kubectl --kubeconfig ${CORE_DEV_KUBECONFIG_PATH} wait --for=condition=Ready --timeout=${timeout} -n certs certificate ${certName}`).code

    if (rc != 0) {
        werft.log(slice, "The certificate never became Ready. We are deleting the certificate so that the next job can create a new one")
        const certificateYAML = exec(`kubectl --kubeconfig ${CORE_DEV_KUBECONFIG_PATH} -n certs get certificate ${certName} -o yaml`, { silent: true }).stdout.trim()
        exec(`kubectl --kubeconfig ${CORE_DEV_KUBECONFIG_PATH} -n certs delete certificate ${certName}`, {slice: slice})
        reportCertificateError({certificateName: certName, certifiateYAML: certificateYAML}).catch((error: Error) => console.error("Failed to send message to Slack", error));
        werft.fail(slice, `Timeout while waiting for certificate readiness after ${timeout}. We have deleted the certificate. Please retry your Werft job. The issue has been reported to the Platform team so they can investigate. Sorry for the inconveneince.`)
    }
}

From what I can see

const rc = exec(`kubectl --kubeconfig ${CORE_DEV_KUBECONFIG_PATH} wait --for=condition=Ready --timeout=${timeout} -n certs certificate ${certName}`).code
... From exec
// exec executes a command and throws an exception if that command exits with a non-zero exit code

will throw an exception if if it fails. Therefore, the code below will never execute. Was this intended?

[meysholdt]

well spotted! exec() needs to have checkRC set to true not throw an exception.

[meysholdt]

#11122 is certainly an improvement, but not a fix yet, because cmctl status certificate <cert> is not used yet.

[meysholdt]

fixed via #11141