Node fails to start with error access to vhost '/' refused for user 'XXXX': vhost '/' is down

[ravisingh-adeptia]

I have deployed rabbitmq as a statefulset on kubernetes cluster, there I am frequently facing error access to vhost '/' refused for user 'XXXX': vhost '/' is down.

Rabbitmq version: 3.9.7 on Erlang 24.1.1 [jit]

Error stack:
[info] <0.508.0> Making sure data directory '/bitnami/rabbitmq/mnesia/adeptia-rabbitmq/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L' for vhost '/' exists
Starting message stores for vhost '/'
2023-08-26 01:45:04.966313+00:00 [info] <0.513.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_transient": using rabbit_msg_store_ets_index to provide index
2023-08-26 01:45:05.149240+00:00 [erro] <0.508.0> Failed to start message store of type msg_store_transient for vhost '/': {{{badmatch,
2023-08-26 01:45:05.149240+00:00 [erro] <0.508.0> {error,
2023-08-26 01:45:05.149240+00:00 [erro] <0.508.0> {"/bitnami/rabbitmq/mnesia/adeptia-rabbitmq/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient",
2023-08-26 01:45:05.149240+00:00 [erro] <0.508.0> eexist}}},
2023-08-26 01:45:05.149240+00:00 [erro] <0.508.0> [{rabbit_msg_store,
2023-08-26 01:45:05.149240+00:00 [erro] <0.508.0> init,
2023-08-26 01:45:05.149240+00:00 [erro] <0.508.0> 1,
2023-08-26 01:45:05.149240+00:00 [erro] <0.508.0> [{file,
2023-08-26 01:45:05.149240+00:00 [erro] <0.508.0> "rabbit_msg_store.erl"},
2023-08-26 01:45:05.149240+00:00 [erro] <0.508.0> {line,
2023-08-26 01:45:05.149240+00:00 [erro] <0.508.0> 724}]},
2023-08-26 01:45:05.149240+00:00 [erro] <0.508.0> {gen_server2,
2023-08-26 01:45:05.149240+00:00 [erro] <0.508.0> init_it,
2023-08-26 01:45:05.149240+00:00 [erro] <0.508.0> 6,
2023-08-26 01:45:05.149240+00:00 [erro] <0.508.0> [{file,
2023-08-26 01:45:05.149240+00:00 [erro] <0.508.0> "gen_server2.erl"},
2023-08-26 01:45:05.149240+00:00 [erro] <0.508.0> {line,
2023-08-26 01:45:05.149240+00:00 [erro] <0.508.0> 565}]},
2023-08-26 01:45:05.149240+00:00 [erro] <0.508.0> {proc_lib,
2023-08-26 01:45:05.149240+00:00 [erro] <0.508.0> init_p_do_apply,
2023-08-26 01:45:05.149240+00:00 [erro] <0.508.0> 3,
2023-08-26 01:45:05.149240+00:00 [erro] <0.508.0> [{file,
2023-08-26 01:45:05.149240+00:00 [erro] <0.508.0> "proc_lib.erl"},
2023-08-26 01:45:05.149240+00:00 [erro] <0.508.0> {line,
2023-08-26 01:45:05.149240+00:00 [erro] <0.508.0> 226}]}]},
2023-08-26 01:45:05.149240+00:00 [erro] <0.508.0> {child,
2023-08-26 01:45:05.149240+00:00 [erro] <0.508.0> undefined,
2023-08-26 01:45:05.149240+00:00 [erro] <0.508.0> msg_store_transient,
2023-08-26 01:45:05.149240+00:00 [erro] <0.508.0> {rabbit_msg_store,
2023-08-26 01:45:05.149240+00:00 [erro] <0.508.0> start_link,
2023-08-26 01:45:05.149240+00:00 [erro] <0.508.0> [msg_store_transient,
2023-08-26 01:45:05.149240+00:00 [erro] <0.508.0> "/bitnami/rabbitmq/mnesia/adeptia-rabbitmq/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L",
2023-08-26 01:45:05.149240+00:00 [erro] <0.508.0> undefined,
2023-08-26 01:45:05.149240+00:00 [erro] <0.508.0> {#Fun<rabbit_variable_queue.0.124157698>,
2023-08-26 01:45:05.149240+00:00 [erro] <0.508.0> ok}]},
2023-08-26 01:45:05.149240+00:00 [erro] <0.508.0> transient,
2023-08-26 01:45:05.149240+00:00 [erro] <0.508.0> 600000,
2023-08-26 01:45:05.149240+00:00 [erro] <0.508.0> worker,
2023-08-26 01:45:05.149240+00:00 [erro] <0.508.0> [rabbit_msg_store]}}
2023-08-26 01:45:05.149443+00:00 [erro] <0.513.0> crasher:
2023-08-26 01:45:05.149443+00:00 [erro] <0.513.0> initial call: rabbit_msg_store:init/1
2023-08-26 01:45:05.149443+00:00 [erro] <0.513.0> pid: <0.513.0>
2023-08-26 01:45:05.149443+00:00 [erro] <0.513.0> registered_name: []
2023-08-26 01:45:05.149443+00:00 [erro] <0.513.0> exception exit: {{badmatch,
2023-08-26 01:45:05.149443+00:00 [erro] <0.513.0> {error,
2023-08-26 01:45:05.149443+00:00 [erro] <0.513.0> {"/bitnami/rabbitmq/mnesia/adeptia-rabbitmq/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient",
2023-08-26 01:45:05.149443+00:00 [erro] <0.513.0> eexist}}},
2023-08-26 01:45:05.149443+00:00 [erro] <0.513.0> [{rabbit_msg_store,init,1,
2023-08-26 01:45:05.149443+00:00 [erro] <0.513.0> [{file,"rabbit_msg_store.erl"},{line,724}]},
2023-08-26 01:45:05.149443+00:00 [erro] <0.513.0> {gen_server2,init_it,6,
2023-08-26 01:45:05.149443+00:00 [erro] <0.513.0> [{file,"gen_server2.erl"},{line,565}]},
2023-08-26 01:45:05.149443+00:00 [erro] <0.513.0> {proc_lib,init_p_do_apply,3,
2023-08-26 01:45:05.149443+00:00 [erro] <0.513.0> [{file,"proc_lib.erl"},{line,226}]}]}
2023-08-26 01:45:05.149443+00:00 [erro] <0.513.0> in function gen_server2:init_it/6 (gen_server2.erl, line 608)
2023-08-26 01:45:05.149443+00:00 [erro] <0.513.0> ancestors: [<0.507.0>,<0.506.0>,rabbit_vhost_sup_sup,rabbit_sup,
2023-08-26 01:45:05.149443+00:00 [erro] <0.513.0> <0.223.0>]
2023-08-26 01:45:05.149443+00:00 [erro] <0.513.0> message_queue_len: 0
2023-08-26 01:45:05.149443+00:00 [erro] <0.513.0> messages: []
2023-08-26 01:45:05.149443+00:00 [erro] <0.513.0> links: [<0.507.0>]
2023-08-26 01:45:05.149443+00:00 [erro] <0.513.0> dictionary: []
2023-08-26 01:45:05.149443+00:00 [erro] <0.513.0> trap_exit: true
2023-08-26 01:45:05.149443+00:00 [erro] <0.513.0> status: running
2023-08-26 01:45:05.149443+00:00 [erro] <0.513.0> heap_size: 2586
2023-08-26 01:45:05.149443+00:00 [erro] <0.513.0> stack_size: 29
2023-08-26 01:45:05.149443+00:00 [erro] <0.513.0> reductions: 3071
2023-08-26 01:45:05.149443+00:00 [erro] <0.513.0> neighbours:
2023-08-26 01:45:05.149443+00:00 [erro] <0.513.0>
2023-08-26 01:45:05.232984+00:00 [erro] <0.508.0> Unable to recover vhost <<"/">> data. Reason {error,
2023-08-26 01:45:05.232984+00:00 [erro] <0.508.0> {{{badmatch,
2023-08-26 01:45:05.232984+00:00 [erro] <0.508.0> {error,
2023-08-26 01:45:05.232984+00:00 [erro] <0.508.0> {"/bitnami/rabbitmq/mnesia/adeptia-rabbitmq/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient",
2023-08-26 01:45:05.232984+00:00 [erro] <0.508.0> eexist}}},
2023-08-26 01:45:05.232984+00:00 [erro] <0.508.0> [{rabbit_msg_store,init,1,
2023-08-26 01:45:05.232984+00:00 [erro] <0.508.0> [{file,
2023-08-26 01:45:05.232984+00:00 [erro] <0.508.0> "rabbit_msg_store.erl"},
2023-08-26 01:45:05.232984+00:00 [erro] <0.508.0> {line,724}]},
2023-08-26 01:45:05.232984+00:00 [erro] <0.508.0> {gen_server2,init_it,6,
2023-08-26 01:45:05.232984+00:00 [erro] <0.508.0> [{file,"gen_server2.erl"},
2023-08-26 01:45:05.232984+00:00 [erro] <0.508.0> {line,565}]},
2023-08-26 01:45:05.232984+00:00 [erro] <0.508.0> {proc_lib,init_p_do_apply,3,
2023-08-26 01:45:05.232984+00:00 [erro] <0.508.0> [{file,"proc_lib.erl"},
2023-08-26 01:45:05.232984+00:00 [erro] <0.508.0> {line,226}]}]},
2023-08-26 01:45:05.232984+00:00 [erro] <0.508.0> {child,undefined,
2023-08-26 01:45:05.232984+00:00 [erro] <0.508.0> msg_store_transient,
2023-08-26 01:45:05.232984+00:00 [erro] <0.508.0> {rabbit_msg_store,start_link,
2023-08-26 01:45:05.232984+00:00 [erro] <0.508.0> [msg_store_transient,
2023-08-26 01:45:05.232984+00:00 [erro] <0.508.0> "/bitnami/rabbitmq/mnesia/adeptia-rabbitmq/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L",
2023-08-26 01:45:05.232984+00:00 [erro] <0.508.0> undefined,
2023-08-26 01:45:05.232984+00:00 [erro] <0.508.0> {#Fun<rabbit_variable_queue.0.124157698>,
2023-08-26 01:45:05.232984+00:00 [erro] <0.508.0> ok}]},
2023-08-26 01:45:05.232984+00:00 [erro] <0.508.0> transient,600000,worker,
2023-08-26 01:45:05.232984+00:00 [erro] <0.508.0> [rabbit_msg_store]}}}
2023-08-26 01:45:05.232984+00:00 [erro] <0.508.0> Stacktrace [{rabbit_variable_queue,do_start_msg_store,4,
2023-08-26 01:45:05.232984+00:00 [erro] <0.508.0> [{file,"rabbit_variable_queue.erl"},
2023-08-26 01:45:05.232984+00:00 [erro] <0.508.0> {line,503}]},
2023-08-26 01:45:05.232984+00:00 [erro] <0.508.0> {rabbit_variable_queue,start_msg_store,3,
2023-08-26 01:45:05.232984+00:00 [erro] <0.508.0> [{file,"rabbit_variable_queue.erl"},
2023-08-26 01:45:05.232984+00:00 [erro] <0.508.0> {line,488}]},
2023-08-26 01:45:05.232984+00:00 [erro] <0.508.0> {rabbit_variable_queue,start,2,
2023-08-26 01:45:05.232984+00:00 [erro] <0.508.0> [{file,"rabbit_variable_queue.erl"},
2023-08-26 01:45:05.232984+00:00 [erro] <0.508.0> {line,479}]},
2023-08-26 01:45:05.232984+00:00 [erro] <0.508.0> {rabbit_priority_queue,start,2,
2023-08-26 01:45:05.232984+00:00 [erro] <0.508.0> [{file,"rabbit_priority_queue.erl"},
2023-08-26 01:45:05.232984+00:00 [erro] <0.508.0> {line,85}]},
2023-08-26 01:45:05.232984+00:00 [erro] <0.508.0> {rabbit_classic_queue,recover,2,
2023-08-26 01:45:05.232984+00:00 [erro] <0.508.0> [{file,"rabbit_classic_queue.erl"},
2023-08-26 01:45:05.232984+00:00 [erro] <0.508.0> {line,124}]},
2023-08-26 01:45:05.232984+00:00 [erro] <0.508.0> {rabbit_queue_type,'-recover/2-fun-2-',4,
2023-08-26 01:45:05.232984+00:00 [erro] <0.508.0> [{file,"rabbit_queue_type.erl"},{line,387}]},
2023-08-26 01:45:05.232984+00:00 [erro] <0.508.0> {maps,fold_1,3,[{file,"maps.erl"},{line,410}]},
2023-08-26 01:45:05.232984+00:00 [erro] <0.508.0> {rabbit_vhost,recover,1,[{file,"rabbit_vhost.erl"},{line,60}]}]
2023-08-26 01:45:05.233769+00:00 [erro] <0.506.0> supervisor: {<0.506.0>,rabbit_vhost_sup_wrapper}
2023-08-26 01:45:05.233769+00:00 [erro] <0.506.0> errorContext: start_error
2023-08-26 01:45:05.233769+00:00 [erro] <0.506.0> reason: {error,
2023-08-26 01:45:05.233769+00:00 [erro] <0.506.0> {{{badmatch,
2023-08-26 01:45:05.233769+00:00 [erro] <0.506.0> {error,
2023-08-26 01:45:05.233769+00:00 [erro] <0.506.0> {"/bitnami/rabbitmq/mnesia/adeptia-rabbitmq/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient",
2023-08-26 01:45:05.233769+00:00 [erro] <0.506.0> eexist}}},
2023-08-26 01:45:05.233769+00:00 [erro] <0.506.0> [{rabbit_msg_store,init,1,
2023-08-26 01:45:05.233769+00:00 [erro] <0.506.0> [{file,"rabbit_msg_store.erl"},{line,724}]},
2023-08-26 01:45:05.233769+00:00 [erro] <0.506.0> {gen_server2,init_it,6,
2023-08-26 01:45:05.233769+00:00 [erro] <0.506.0> [{file,"gen_server2.erl"},{line,565}]},
2023-08-26 01:45:05.233769+00:00 [erro] <0.506.0> {proc_lib,init_p_do_apply,3,
2023-08-26 01:45:05.233769+00:00 [erro] <0.506.0> [{file,"proc_lib.erl"},{line,226}]}]},
2023-08-26 01:45:05.233769+00:00 [erro] <0.506.0> {child,undefined,msg_store_transient,
2023-08-26 01:45:05.233769+00:00 [erro] <0.506.0> {rabbit_msg_store,start_link,
2023-08-26 01:45:05.233769+00:00 [erro] <0.506.0> [msg_store_transient,
2023-08-26 01:45:05.233769+00:00 [erro] <0.506.0> "/bitnami/rabbitmq/mnesia/adeptia-rabbitmq/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L",
2023-08-26 01:45:05.233769+00:00 [erro] <0.506.0> undefined,
2023-08-26 01:45:05.233769+00:00 [erro] <0.506.0> {#Fun<rabbit_variable_queue.0.124157698>,ok}]},
2023-08-26 01:45:05.233769+00:00 [erro] <0.506.0> transient,600000,worker,
2023-08-26 01:45:05.233769+00:00 [erro] <0.506.0> [rabbit_msg_store]}}}
2023-08-26 01:45:05.233769+00:00 [erro] <0.506.0> offender: [{pid,undefined},
2023-08-26 01:45:05.233769+00:00 [erro] <0.506.0> {id,rabbit_vhost_process},
2023-08-26 01:45:05.233769+00:00 [erro] <0.506.0> {mfargs,{rabbit_vhost_process,start_link,[<<"/">>]}},
2023-08-26 01:45:05.233769+00:00 [erro] <0.506.0> {restart_type,permanent},
2023-08-26 01:45:05.233769+00:00 [erro] <0.506.0> {significant,false},
2023-08-26 01:45:05.233769+00:00 [erro] <0.506.0> {shutdown,300000},
2023-08-26 01:45:05.233769+00:00 [erro] <0.506.0> {child_type,worker}]
2023-08-26 01:45:05.233769+00:00 [erro] <0.506.0>
2023-08-26 01:45:05.233645+00:00 [erro] <0.508.0> crasher:
2023-08-26 01:45:05.233645+00:00 [erro] <0.508.0> initial call: rabbit_vhost_process:init/1
2023-08-26 01:45:05.233645+00:00 [erro] <0.508.0> pid: <0.508.0>
2023-08-26 01:45:05.233645+00:00 [erro] <0.508.0> registered_name: []
2023-08-26 01:45:05.233645+00:00 [erro] <0.508.0> exception exit: {error,
2023-08-26 01:45:05.233645+00:00 [erro] <0.508.0> {{{badmatch,
2023-08-26 01:45:05.233645+00:00 [erro] <0.508.0> {error,
2023-08-26 01:45:05.233645+00:00 [erro] <0.508.0> {"/bitnami/rabbitmq/mnesia/adeptia-rabbitmq/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient",
2023-08-26 01:45:05.233645+00:00 [erro] <0.508.0> eexist}}},
2023-08-26 01:45:05.233645+00:00 [erro] <0.508.0> [{rabbit_msg_store,init,1,
2023-08-26 01:45:05.233645+00:00 [erro] <0.508.0> [{file,"rabbit_msg_store.erl"},{line,724}]},
2023-08-26 01:45:05.233645+00:00 [erro] <0.508.0> {gen_server2,init_it,6,
2023-08-26 01:45:05.233645+00:00 [erro] <0.508.0> [{file,"gen_server2.erl"},{line,565}]},
2023-08-26 01:45:05.233645+00:00 [erro] <0.508.0> {proc_lib,init_p_do_apply,3,
2023-08-26 01:45:05.233645+00:00 [erro] <0.508.0> [{file,"proc_lib.erl"},{line,226}]}]},
2023-08-26 01:45:05.233645+00:00 [erro] <0.508.0> {child,undefined,msg_store_transient,
2023-08-26 01:45:05.233645+00:00 [erro] <0.508.0> {rabbit_msg_store,start_link,
2023-08-26 01:45:05.233645+00:00 [erro] <0.508.0> [msg_store_transient,
2023-08-26 01:45:05.233645+00:00 [erro] <0.508.0> "/bitnami/rabbitmq/mnesia/adeptia-rabbitmq/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L",
2023-08-26 01:45:05.233645+00:00 [erro] <0.508.0> undefined,
2023-08-26 01:45:05.233645+00:00 [erro] <0.508.0> {#Fun<rabbit_variable_queue.0.124157698>,
2023-08-26 01:45:05.233645+00:00 [erro] <0.508.0> ok}]},
2023-08-26 01:45:05.233645+00:00 [erro] <0.508.0> transient,600000,worker,
2023-08-26 01:45:05.233645+00:00 [erro] <0.508.0> [rabbit_msg_store]}}}
2023-08-26 01:45:05.233645+00:00 [erro] <0.508.0> in function gen_server2:init_it/6 (gen_server2.erl, line 600)
2023-08-26 01:45:05.233645+00:00 [erro] <0.508.0> ancestors: [<0.506.0>,rabbit_vhost_sup_sup,rabbit_sup,<0.223.0>]
2023-08-26 01:45:05.233645+00:00 [erro] <0.508.0> message_queue_len: 0
2023-08-26 01:45:05.233645+00:00 [erro] <0.508.0> messages: []
2023-08-26 01:45:05.233645+00:00 [erro] <0.508.0> links: [<0.506.0>]
2023-08-26 01:45:05.233645+00:00 [erro] <0.508.0> dictionary: []
2023-08-26 01:45:05.233645+00:00 [erro] <0.508.0> trap_exit: true
2023-08-26 01:45:05.233645+00:00 [erro] <0.508.0> status: running
2023-08-26 01:45:05.233645+00:00 [erro] <0.508.0> heap_size: 28690
2023-08-26 01:45:05.233645+00:00 [erro] <0.508.0> stack_size: 29
2023-08-26 01:45:05.233645+00:00 [erro] <0.508.0> reductions: 82553
2023-08-26 01:45:05.233645+00:00 [erro] <0.508.0> neighbours:
2023-08-26 01:45:05.233645+00:00 [erro] <0.508.0>
2023-08-26 01:45:05.235013+00:00 [warn] <0.223.0> Unable to initialize vhost data store for vhost '/'. The vhost will be stopped for this node. Reason: {shutdown, [rabbit_msg_store]}}}}}
2023-08-26 01:45:05.536729+00:00 [info] <0.527.0> Server startup complete; 8 plugins started.
2023-08-26 01:45:05.536729+00:00 [info] <0.527.0> * rabbitmq_prometheus
2023-08-26 01:45:05.536729+00:00 [info] <0.527.0> * rabbitmq_peer_discovery_k8s
2023-08-26 01:45:05.536729+00:00 [info] <0.527.0> * rabbitmq_peer_discovery_common
2023-08-26 01:45:05.536729+00:00 [info] <0.527.0> * rabbitmq_auth_backend_ldap
2023-08-26 01:45:05.536729+00:00 [info] <0.527.0> * rabbitmq_delayed_message_exchange
2023-08-26 01:45:05.536729+00:00 [info] <0.527.0> * rabbitmq_management
2023-08-26 01:45:05.536729+00:00 [info] <0.527.0> * rabbitmq_web_dispatch
2023-08-26 01:45:05.536729+00:00 [info] <0.527.0> * rabbitmq_management_agent
2023-08-26 01:45:31.242987+00:00 [info] <0.713.0> accepting AMQP connection <0.713.0> (10.0.4.250:33358 -> 10.0.4.12:5671)
2023-08-26 01:45:31.245519+00:00 [info] <0.713.0> Connection <0.713.0> (10.0.4.250:33358 -> 10.0.4.12:5671) has a client-provided name: rabbitConnectionFactory#7bcc0392:22
2023-08-26 01:45:31.246976+00:00 [erro] <0.713.0> Error on AMQP connection <0.713.0> (10.0.4.250:33358 -> 10.0.4.12:5671 - rabbitConnectionFactory#7bcc0392:22, vhost: 'none', user: 'adeptia', state: opening), channel 0:
2023-08-26 01:45:31.246976+00:00 [erro] <0.713.0> {handshake_error,opening,
2023-08-26 01:45:31.246976+00:00 [erro] <0.713.0> {amqp_error,internal_error,
2023-08-26 01:45:31.246976+00:00 [erro] <0.713.0> "access to vhost '/' refused for user 'adeptia': vhost '/' is down",
2023-08-26 01:45:31.246976+00:00 [erro] <0.713.0> 'connection.open'}}
2023-08-26 01:45:31.247628+00:00 [info] <0.713.0> closing AMQP connection <0.713.0> (10.0.4.250:33358 -> 10.0.4.12:5671 - rabbitConnectionFactory#7bcc0392:22, vhost: 'none', user: 'adeptia')
2023-08-26 01:45:31.254313+00:00 [info] <0.724.0> accepting AMQP connection <0.724.0> (10.0.4.250:33360 -> 10.0.4.12:5671)

On restarting the pod/container the issue is resolved.
But on production cluster user doesn't have privileges to restart the pod/container.
Also I want to understand why I am getting this error.

[michaelklishin]

RabbitMQ 3.9 has reached EOL. You will not get any further support from the core team unless you upgrade to the latest supported version.

Unable to initialize vhost data store for vhost '/'

is the log line you are looking for. Something prevents a message store from starting in the first boot. That something is an environment-specific problem, maybe the storage device is not ready initially on pod startup, or something like that.

[ravisingh-adeptia]

@michaelklishin Which version should I consider for upgrading rabbitmq

[michaelklishin]

The latest there is.

See Upgrading RabbitMQ and Blue-Green Deployment before you try to jump straight from 3.9 to 3.12.10.

[michaelklishin]

I recall discussing this with another core team member and our conclusion was the following: if the mounted volume is not yet ready for writes by the time the node boots, it will fail to seed the data and the virtual host then will fail to stop.

This is pretty clearly hinted at by one of the function names: rabbit_variable_queue:do_start_msg_store/4.

We have never seen this behavior outside of Kubernetes, and RabbitMQ nodes do not do anything creative when it comes to initializing the schema data store or the CQ message store. So there is nothing to "fix once and for all" in RabbitMQ.

A while ago we have considered adding optional delays before and after the node boots, for very different reasons. The former might help here. Or you can inject a startup pause using a Kubernetes-specific method, e.g. an init container that would verify that the volume is ready (writeable).

To our knowledge, this behavior was never reported by those who use our Kubernetes Cluster Operator. Most likely because it introduces a startup delay to work around a widely known unfortunate CoreDNS caching behavior/default.

So, a similar delay will likely help with volumes not being ready early enough.

[bdoublet91]

Hi, thanks for your reply. I didnt check discussion topic.
For now, I am testing a fix that check rabbitmq data dir is writtable before start rabbitmq in the container:

#! /bin/bash

RABBITMQ_DATA_DIR="/var/lib/rabbitmq"
TEST_FILE="$RABBITMQ_DATA_DIR/.writetest"

echo "Checking if RabbitMQ volume is writable..."
SUCCESS="false"
# Try writing a temp file, retry if not writable
for i in $(seq 1 30); do
  if touch "$TEST_FILE" 2>/dev/null; then
    echo "Writable: $RABBITMQ_DATA_DIR"
    rm "$TEST_FILE"
    SUCCESS="true"
    break
  else
    echo "Attempt $i: Volume not writable yet, retrying in 2s..."
    sleep 2
  fi
done

if [[ "${SUCCESS}" == "false" ]]; then
  echo "ERROR: Volume $RABBITMQ_DATA_DIR is not writable after timeout."
  exit 1
fi

/usr/local/bin/docker-entrypoint.sh rabbitmq-server

I have also change the mountdir from /var/lib/rabbitmq/mnesia to /var/lib/rabbitmq and bind to my NFS on the docker host.
Also change the update workflow, stop the container before start the new one, maybe concurrency access could lead to corrupted data

    deploy:
      update_config:
        order: stop-first

I will try this week these fix and I will give you a feedback.
thanks