2.7.0 doesn't create connection string secret anymore

[fe-ax]

What did you do to encounter the bug?
Steps to reproduce the behaviour: Launch a new cluster using the operator, add a AtlasDatabaseUser

What did you expect?
A secret with the connection string like this:

apiVersion: v1
data:
  connectionStringPrivate: ""
  connectionStringPrivateSrv: ""
  connectionStringStandard: asdasd=
  connectionStringStandardSrv: asdasd=
  password: asdasd=
  username: asdasd=
kind: Secret
metadata:
  creationTimestamp: "2025-01-31T09:33:31Z"
  labels:
    atlas.mongodb.com/cluster-name: pr-frontend-14-fb7f3fd
    atlas.mongodb.com/project-id: 66c...b2fdf4
    atlas.mongodb.com/type: credentials
  name: vi...nting-dotnet
  namespace: pr-frontend-1456
  resourceVersion: "266633950"
  uid: fcb5...2226986
type: Opaque

What happened instead?
No secret is created like in 2.6.1

Operator Information

• 2.7.0

Kubernetes Cluster Information

• eks
• 1.30

{"level":"INFO","time":"2025-02-10T15:54:01.618Z","msg":"-> Starting AtlasDatabaseUser reconciliation","spec":{"projectRef":{"name":"mongodb-atlas-project","namespace":"mongodb-atlas"},"databaseName":"admin","roles":[{"roleName":"readWrite","databaseName":"configs"}],"scopes":[{"name":"pr-shipping-69-65f6a1b","type":"CLUSTER"}],"passwordSecretRef":{"name":"pr-shipping-69-65f6a1b-configs-password"},"username":"pr-shipping-69-65f6a1b-configs-dotnet","oidcAuthType":"NONE","awsIamType":"NONE","x509Type":"NONE"},"status":{"conditions":[{"type":"Ready","status":"False","lastTransitionTime":"2025-02-10T15:53:39Z"},{"type":"ResourceVersionIsValid","status":"True","lastTransitionTime":"2025-02-10T15:53:39Z"},{"type":"ValidationSucceeded","status":"True","lastTransitionTime":"2025-02-10T15:53:39Z"},{"type":"DatabaseUserReady","status":"False","lastTransitionTime":"2025-02-10T15:53:40Z","reason":"DeploymentAppliedDatabaseUsersChanges","message":"Clusters are scheduled to handle database users updates"}],"observedGeneration":1,"passwordVersion":"298205977","name":"pr-shipping-69-65f6a1b-configs-dotnet"}}
{"level":"INFO","time":"2025-02-10T15:54:09.734Z","msg":"-> Starting AtlasDatabaseUser reconciliation","spec":{"projectRef":{"name":"mongodb-atlas-project","namespace":"mongodb-atlas"},"databaseName":"admin","roles":[{"roleName":"readWrite","databaseName":"scheduler"}],"scopes":[{"name":"pr-shipping-69-65f6a1b","type":"CLUSTER"}],"passwordSecretRef":{"name":"pr-shipping-69-65f6a1b-scheduler-password"},"username":"pr-shipping-69-65f6a1b-scheduler-dotnet","oidcAuthType":"NONE","awsIamType":"NONE","x509Type":"NONE"},"status":{"conditions":[{"type":"Ready","status":"True","lastTransitionTime":"2025-02-10T15:53:46Z"},{"type":"ResourceVersionIsValid","status":"True","lastTransitionTime":"2025-02-10T15:53:34Z"},{"type":"ValidationSucceeded","status":"True","lastTransitionTime":"2025-02-10T15:53:34Z"},{"type":"DatabaseUserReady","status":"True","lastTransitionTime":"2025-02-10T15:53:46Z"}],"observedGeneration":1,"passwordVersion":"298205652","name":"pr-shipping-69-65f6a1b-scheduler-dotnet"}}
{"level":"INFO","time":"2025-02-10T15:54:13.399Z","msg":"-> Starting AtlasDatabaseUser reconciliation","spec":{"projectRef":{"name":"mongodb-atlas-project","namespace":"mongodb-atlas"},"databaseName":"admin","roles":[{"roleName":"readWrite","databaseName":"printing"}],"scopes":[{"name":"pr-shipping-69-65f6a1b","type":"CLUSTER"}],"passwordSecretRef":{"name":"pr-shipping-69-65f6a1b-printing-password"},"username":"pr-shipping-69-65f6a1b-printing-dotnet","oidcAuthType":"NONE","awsIamType":"NONE","x509Type":"NONE"},"status":{"conditions":[{"type":"Ready","status":"True","lastTransitionTime":"2025-02-10T15:53:50Z"},{"type":"ResourceVersionIsValid","status":"True","lastTransitionTime":"2025-02-10T15:53:35Z"},{"type":"ValidationSucceeded","status":"True","lastTransitionTime":"2025-02-10T15:53:35Z"},{"type":"DatabaseUserReady","status":"True","lastTransitionTime":"2025-02-10T15:53:50Z"}],"observedGeneration":1,"passwordVersion":"298205769","name":"pr-shipping-69-65f6a1b-printing-dotnet"}}

[fe-ax]

The weird thing is that if we delete the secret on an old cluster, which is hanging on the issue with the serverless/flex shim, it will work. If we create a new cluster (flex) it doesn't anymore.

[helderjs]

Hi @fe-ax!
Thanks for the report. We are actively looking into the issue.

[fe-ax]

{"level":"DEBUG","time":"2025-02-10T16:21:21.642Z","msg":"HTTP Request (GET) https://cloud.mongodb.com/api/atlas/v2/groups/66c...df4/flexClusters/pr-shipping-69-65f6a1b [time (ms): 247, status: 200]"}
{"level":"DEBUG","time":"2025-02-10T16:21:21.891Z","msg":"Creating a connection Secret","atlasdeployment":"pr-...-698/atlas","data":{"DBUserName":"pr-...-69-65f6a1b-ftp-dotnet","Password":"","ConnURL":"","SrvConnURL":"mongodb+srv://pr-shipping-69-65f6a1b.....mongodb.net","PrivateConnURLs":null}}
....
{"level":"DEBUG","time":"2025-02-10T16:21:23.255Z","msg":"Creating a connection Secret","atlasdeployment":"pr-...-698/atlas","data":{"DBUserName":"pr-shipping-69-65f6a1b-...-dotnet","Password":"","ConnURL":"","SrvConnURL":"mongodb+srv://pr-shipping-69-65f6a1b......mongodb.net","PrivateConnURLs":null}}
{"level":"DEBUG","time":"2025-02-10T16:21:23.264Z","msg":"Connection Secrets were created/updated: ...-testing-pr-shipping-69-65f6a1b-pr-shipping-69-65f6a1b-....-dotnet, ...-testing-pr-shipping-69-65f6a1b-pr-shipping-698...., ...-testing-pr-shipping-69-65f6a1b-pr-shipping-69-65f6a1b-....-..., ...-testing-pr-shipping-69-65f6a1b-pr-shipping-69-65f6a1b-scheduler-dotnet, ...-testing-pr-shipping-69-65f6a1b-pr-shipping-69-65f6a1b-...-dotnet, ...-testing-pr-shipping-69-65f6a1b-pr-shipping-69-65f6a1b-configs-dotnet","type":"Normal","object":{"kind":"AtlasDeployment","namespace":"pr-...-698","name":"atlas","uid":"d5eed461-4d2d-485b-83e8-8789a2aae5fe","apiVersion":"atlas.mongodb.com/v1","resourceVersion":"298203781"},"reason":"ConnectionSecretsEnsured"}

It says it does, but it doesn't

[helderjs]

Hi @fe-ax

We've been able to reproduce the error and have confirmed the bug. The operator is not handling properly the connection secrets of database user for existing Flex clusters. A temporary solution would be to force the reconciliation of Flex deployment to get it to generate the connection secrets. We are actively working in a solution to be release in upcoming release.

[roothorp]

He @fe-ax - we have a fix for this coming in our next release (2.7.1).

[fe-ax]

Thanks. We spend most of the day learning to write a fix using Kyverno. It might be good to do a thorough scan of all the dedicated and serverless calls to find what else might be missed.

These two issues caused a major issue. We couldn't move forward, we couldn't move backwards. We test our setup using the Kubernetes e2e testing framework, but as the shim was put in place on your side there was no way to prevent this.

[vmanikes]

I agree with @fe-ax . With every new release there is a new issue with the operator. Honestly I am scared bumping version of the operator these days

[dan-mckean]

Hi, I wanted to reply just to apologize for the impact here. While our teams did a lot of work to ensure the shimming worked as expected in Atlas APIs, there was considerable complexity there. There was a slight (now resolved) misalignment, and the change coming soon in a new release of the Operator (2.7.1) will help address the problems experienced.

[s-urbaniak]

Closing as this is fixed in the current 2.7.1 release.