[Suggestion] A limit on the number of internal database users that can be created (by accident)

[ohasi]

RabbitMQ series

4.0.x

Operating system (distribution) used

Debian 9

How is RabbitMQ deployed?

Community Docker image

What would you like to suggest for a future version of RabbitMQ?

We have a RabbitMQ instance that is used by many services.
In our development environment, we often encounter bugs in our own services that cause a very big amount of users to be created in the RabbitMQ instance .

In the documentation users are described as a "cheap" resource, but in practice, when 500,000+ users are created the RabbitMQ management API starts returning 5xx when trying to perform any operation that has to do with user management.

Right now, our only solution is to write a script that deletes users one by one using the rabbitmqctl delete_user command, but it takes days to delete this many users when deleting one by one.

Any of the following features would be a great help:

1. A rabbitmqctl command that deletes all users with a name matching a pattern
2. A configuration option to limit the amount of users that can be created across the entire RabbitMQ instance
3. Some sort of TTL mechanism for users, that can automatically delete unused users.

[michaelklishin]

@ohasi options 1 and 3 are guaranteed to backfire.

We cannot save users with automation tools from themselves. Yes, there are virtual host limits but they limit access to a very different kind of resources:

1. Those that can be impossible or very expensive to re-create (a virtual host)
2. Those that we know some applications will inevitably leak, given a long enough lifetime of a cluster (queues, streams)

I am not against having a limit on the number of users but the whole situation is a bit silly. Deployment automation can create 500K stray users in your org? That sounds highly unusual.

Perhaps try switching to JWT tokens, a custom HTTP authentication service plus the caching backend, or LDAP. They all offload the "too many internal database users" problem to external tools.

[michaelklishin]

My colleagues recommend using development or staging clusters/environments for testing out automation over modifying RabbitMQ. It's difficult to argue with that TBH.

[ohasi]

We 100% have a problem, and we are working to solve it in means that have nothing to do with rabbitmq itself.
Even if there was already a feature that enables limiting the amount of users, it would not solve our bug and that's not the purpose.

I just believe that setting a limit for the amount of users in a rabbitmq instance (that limit can even be hard coded) to protect the rabbitmq server from misuse would marginally improve reliability, but I agree this is a pretty obscure case.
I don't think bulk deleting users is an obscure usecase though (I might be wrong). And I do believe adding a command or two to allow easier management of rabbitmq users when the need arises.
Again - it's obvious that if a user created 500k users, the problem lies with the user. We have obviously taken immediate steps to stop the problem. But - we were left with 500k users to delete, which there is no good way to do right now.

[ohasi]

My colleagues recommend using development or staging clusters/environments for testing out automation over modifying RabbitMQ. It's difficult to argue with that TBH.

We experienced the issue in a development environment, but returning the environment to a usable state was difficult and sparked some thoughts about how our experience can be used to improve rabbitmq

[lukebakken]

we were left with 500k users to delete, which there is no good way to do right now.

In a dev / test environment, rm -rf /var/lib/rabbitmq/* (i.e. nuking the data on each node) should be acceptable. Just a suggestion.

[michaelklishin]

And you can export definitions except for the users, with rabbitmqadmin v2:

rabbitmqadmin definitions export --transformations exclude_users

then you can wipe the node as @lukebakken explains, and reimport the definitions that include everything except for the users.

[ohasi]

This solution worked pretty well, with some caveats:

Our deployment (using rabbitmq operator, I'm out of office so I can't recall which version of the operator) deployed pods with rabbitmqadmin v1, so exporting definitions without users was not possible.
We'll upgrade our clusters .

I still believe a global user limit is a good idea, but that's not for me to decide.

Anyway - thanks for the quick and helpful responses. I honestly didn't expect it to be this quick, and the experience reinforces my opinion of rabbitmq as a good, reliable product.

[MirahImage]

rabbitmqadmin v2 is a standalone binary, it doesn't need to be part of the pod. It can be run from anywhere with access to the management API. https://github.com/rabbitmq/rabbitmqadmin-ng