CLOUDP-266544: Support local resource credentials

[josvazg]

• Add common infrastructure for adding local per CRD credentials at pkg/api/credentials.go.
• Use the new infra from AtlasDatabaseUser's CRD and Controller.
• Solve whether the credentials are taken from the current resource, the project or render nil.

Testing:

• Unit tests on new ComputeSecret.
• Conversion test fix: credentials must be ignored on comparisons between the Atlas and Kubernetes Spec.
• Users e2e test: Second user takes locally defined credentials. Also if local credentials do not exist, reconciliation fails as expected.

All Submissions:

• Have you signed our CLA?

[github-actions]

https://app.codecov.io/github/mongodb/mongodb-atlas-kubernetes/commit/6450dea0900bd5a11877c997411047623a581882

[josvazg]

One nit about the global secret fallback handling. The logic is quite confusing to the reader. Why not make it explicit, where:

type Provider interface {
...
	GlobalSecret() client.ObjectKey
}

and then just invoke:

credentialsSecret, err := customresource.ComputeSecret(r.AtlasProvider.GlobalSecret(), project, databaseUser)
...

where the fallback logic is explicit in ComputeSecret:

func ComputeSecret(globalSecret client.ObjectKey, project *akov2.AtlasProject, resource api.ResourceWithCredentials) (*client.ObjectKey, error) {
	if globalSecret.Name != "" {
		return &globalSecret, nil
	}
	return resolveConnectionSecret(project, resource)
}

There is less indirection and makes the fallback explicit.

In the end I removed the global fallback from this part of the resolution. That will still work as before.

The only loss is we do not have precise errors when there is no global fallback, but that scenario was problematic to open up so soon anyway. We should decide if the global fallback will be optional separately.

[igor-karpukhin]

@josvazg Why do we need to duplicate API credentials?

[josvazg]

You are right, we do not need this credentials.

I can simplify, but then the gov test issues will get me, so I won't be able to merge today.

[josvazg]

removed

[s-urbaniak]

rerunning e2e gov, let's evaluate if this is still a blocker.

[josvazg]

Our nightly test suite saw recovery today at midnight on the gov tests:
✅ cloud-tests / test-e2e-gov / E2E Gov tests

[josvazg]

@s-urbaniak all green again, I closed CLOUDP-270948