Queue process termination and restart causes consumer to stop receiving messages

[djjeane]

Describe the bug

We are noticing some odd behavior when our queue processes crash. The reason for the crashing is hard to identify, we are assuming some windows security software is interacting poorly with Rabbit.

When the queues do crash, our consumers tied to those queues stop receiving messages. When the queues automatically restart the consumers still do not receive messages. The channels used by these consumers do not encounter a ChannelShutdown event, so our software is not detecting that anything has gone wrong with the rabbit process.

Snippet of Queue crash stack:

2025-01-22 12:01:04.367000-06:00 [error] <0.1774.0> ** Reason for termination == 2025-01-22 12:01:04.367000-06:00 [error] <0.1774.0> ** {{badmatch,{error,eacces}}, 2025-01-22 12:01:04.367000-06:00 [error] <0.1774.0> [{rabbit_classic_queue_index_v2,new_segment_file,3, 2025-01-22 12:01:04.367000-06:00 [error] <0.1774.0> [{file,"rabbit_classic_queue_index_v2.erl"},{line,594}]}, 2025-01-22 12:01:04.367000-06:00 [error] <0.1774.0> {rabbit_classic_queue_index_v2,publish,8, 2025-01-22 12:01:04.367000-06:00 [error] <0.1774.0> [{file,"rabbit_classic_queue_index_v2.erl"},{line,583}]}, 2025-01-22 12:01:04.367000-06:00 [error] <0.1774.0> {rabbit_variable_queue,maybe_write_index_to_disk,3, 2025-01-22 12:01:04.367000-06:00 [error] <0.1774.0> [{file,"rabbit_variable_queue.erl"},{line,1841}]}, 2025-01-22 12:01:04.367000-06:00 [error] <0.1774.0> {rabbit_variable_queue,publish_delivered1,5, 2025-01-22 12:01:04.367000-06:00 [error] <0.1774.0> [{file,"rabbit_variable_queue.erl"},{line,1736}]}, 2025-01-22 12:01:04.367000-06:00 [error] <0.1774.0> {rabbit_variable_queue,publish_delivered,4, 2025-01-22 12:01:04.367000-06:00 [error] <0.1774.0> [{file,"rabbit_variable_queue.erl"},{line,542}]}, 2025-01-22 12:01:04.367000-06:00 [error] <0.1774.0> {rabbit_priority_queue,publish_delivered,4, 2025-01-22 12:01:04.367000-06:00 [error] <0.1774.0> [{file,"rabbit_priority_queue.erl"},{line,219}]}, 2025-01-22 12:01:04.367000-06:00 [error] <0.1774.0> {rabbit_amqqueue_process,'-attempt_delivery/4-fun-0-',10, 2025-01-22 12:01:04.367000-06:00 [error] <0.1774.0> [{file,"rabbit_amqqueue_process.erl"},{line,680}]}, 2025-01-22 12:01:04.367000-06:00 [error] <0.1774.0> {rabbit_queue_consumers,deliver_to_consumer,4, 2025-01-22 12:01:04.367000-06:00 [error] <0.1774.0> [{file,"rabbit_queue_consumers.erl"},{line,344}]}]} 2025-01-22 12:01:04.367000-06:00 [error] <0.1774.0> 2025-01-22 12:01:04.367000-06:00 [error] <0.1774.0> crasher: 2025-01-22 12:01:04.367000-06:00 [error] <0.1774.0> initial call: rabbit_amqqueue_process:init/1 2025-01-22 12:01:04.367000-06:00 [error] <0.1774.0> pid: <0.1774.0> 2025-01-22 12:01:04.367000-06:00 [error] <0.1774.0> registered_name: [] 2025-01-22 12:01:04.367000-06:00 [error] <0.1774.0> exception exit: {{badmatch,{error,eacces}}, 2025-01-22 12:01:04.367000-06:00 [error] <0.1774.0> [{rabbit_classic_queue_index_v2,new_segment_file,3, 2025-01-22 12:01:04.367000-06:00 [error] <0.1774.0> [{file,"rabbit_classic_queue_index_v2.erl"}, 2025-01-22 12:01:04.367000-06:00 [error] <0.1774.0> {line,594}]}, 2025-01-22 12:01:04.367000-06:00 [error] <0.1774.0> {rabbit_classic_queue_index_v2,publish,8, 2025-01-22 12:01:04.367000-06:00 [error] <0.1774.0> [{file,"rabbit_classic_queue_index_v2.erl"}, 2025-01-22 12:01:04.367000-06:00 [error] <0.1774.0> {line,583}]}, 2025-01-22 12:01:04.367000-06:00 [error] <0.1774.0> {rabbit_variable_queue,maybe_write_index_to_disk,3, 2025-01-22 12:01:04.367000-06:00 [error] <0.1774.0> [{file,"rabbit_variable_queue.erl"},{line,1841}]}, 2025-01-22 12:01:04.367000-06:00 [error] <0.1774.0> {rabbit_variable_queue,publish_delivered1,5, 2025-01-22 12:01:04.367000-06:00 [error] <0.1774.0> [{file,"rabbit_variable_queue.erl"},{line,1736}]}, 2025-01-22 12:01:04.367000-06:00 [error] <0.1774.0> {rabbit_variable_queue,publish_delivered,4, 2025-01-22 12:01:04.367000-06:00 [error] <0.1774.0> [{file,"rabbit_variable_queue.erl"},{line,542}]}, 2025-01-22 12:01:04.367000-06:00 [error] <0.1774.0> {rabbit_priority_queue,publish_delivered,4, 2025-01-22 12:01:04.367000-06:00 [error] <0.1774.0> [{file,"rabbit_priority_queue.erl"},{line,219}]}, 2025-01-22 12:01:04.367000-06:00 [error] <0.1774.0> {rabbit_amqqueue_process,'-attempt_delivery/4-fun-0-', 2025-01-22 12:01:04.367000-06:00 [error] <0.1774.0> 10, 2025-01-22 12:01:04.367000-06:00 [error] <0.1774.0> [{file,"rabbit_amqqueue_process.erl"},{line,680}]}, 2025-01-22 12:01:04.367000-06:00 [error] <0.1774.0> {rabbit_queue_consumers,deliver_to_consumer,4, 2025-01-22 12:01:04.367000-06:00 [error] <0.1774.0> [{file,"rabbit_queue_consumers.erl"},{line,344}]}]} 2025-01-22 12:01:04.367000-06:00 [error] <0.1774.0> in function gen_server2:terminate/3 (gen_server2.erl, line 1172) 2025-01-22 12:01:04.367000-06:00 [error] <0.1774.0> ancestors: [<0.1773.0>,<0.582.0>,<0.560.0>,<0.559.0>, 2025-01-22 12:01:04.367000-06:00 [error] <0.1774.0> rabbit_vhost_sup_sup,rabbit_sup,<0.211.0>] 2025-01-22 12:01:04.367000-06:00 [error] <0.1774.0> message_queue_len: 186

Snippet of Queue Restart
2025-01-22 12:01:04.538000-06:00 [error] <0.3464.0> Restarting crashed queue 'queue_1' in vhost '/'. 2025-01-22 12:01:04.538000-06:00 [warning] <0.3464.0> Queue queue_1 in vhost / dropped 0/0/0 persistent messages and 0 transient messages after unclean shutdown

The below link seems to be a related issue, and the guidance was to check in on security software. We have done this to the best of our ability by disabling the security software that was exposed to use, but we don't fundamentally control the host, so its possible that other security software is interacting with RabbitMQ.
Link to seemingly related issue

Any guidance on:

• How to detect when these crashes are occurring
• How to cleanly recover from this situation
• How to avoid the crashes in the first place.

Reproduction steps

1.Have a consumer with a channel that is bound to Queue A
2.Causes Queue A to have an unclean shutdown
3.Witness channel shutdown not be fired
4.Allow Queue A to restart
5.Witness consumer fail to process messages from Queue A until connection/channel are destroyed and recreated

Expected behavior

• I would expect when a queue process crashes, the channel on the consumer/publisher would be shutdown as well.
• If the channels would not be shutdown, I would expect message delivery to continue once the queue process has restarted

Additional context

• All queues involved are classic
• All messages sent are persistent
• Initial queue crashes always involve a specific type of consumer (worker based architecture)
• We do not have control of the security software running on the Windows host, so its possible that security software is causing the crashes, we are still tasked with recovering from this issue.

[djjeane]

Versions:
RabbitMQ: 4.0.4
Erlang 27.1.2

[michaelklishin]

@djjeane

{{badmatch,{error,eacces}}

is the line you are looking for. A filesystem operation has failed with eacces, which has caused a message store index exception.

Queue replicas and channels are completely independent from each other, although channels can monitor queues.

Channels do retry some operations with a delay when a queue does not have an elected leader (in the case of quorum queues and streams). So for non-transient data, use one of those two.

For transient (specifically exclusive, since non-exclusive non-durable queues are going away later in the 4.x series), there already is #12949 which is not as trivial to address as it may sound.

[michaelklishin]

A highly related protocol extension: consumer cancellation notifications.

A failure is not the same scenario as a deletion, because in the latter case usually (sadly far from always) a queue with the same name is unlikely to re-appear in the short term. With failures, you never know because that depends entirely on the underlying cause. The eacces I/O exception can last for hours, for example, if it has to do with file permissions and not an intermittent condition some (usually netowrk-attached/distributed) storage systems may report as eacces.

[djjeane]

Okay gotcha, thanks for the response.

Yeah we had identified this as an issue stemming from a filesystem operation. Resolving why we are getting filesystem errors has been painful thus far.

So when using classic queues, this behavior is expected is seems. So to get some clarity, if we moved to using quorum queues, would our channels then be able to tolerate/detect message store index exceptions.

Just some added detail, we are not utilizing clustering of our rabbitmq server. We have a single instance of the RabbitMQ Server. Our classic queues are durable and non-exclusive.

Thanks again for the swift response!

[djjeane]

Also, you mentioned this consumer cancellation extension. Do you think that would this message store index exception and allow us to respond to that? Im pretty fresh to rabbit, so im not following the failure/deletion comment you made.

Also you are a saint haha

[michaelklishin]

Consumer cancellation notifications were introduced to allow applications observe queue deletion. I suspect they have never been adopted for quorum queues and streams, which may change as part of #12949. The parts that could have sent them for CMQs are gone together with CMQs in 4.x.

[klettier]

Hi guys,

This issue appears to be related to what we previously encountered here.

We still experience this problem occasionally, but only on Windows. It seems to be caused by an unknown antivirus blocking our process, leading to consumer crashes with no recovery options.

Does anyone know of a reference page listing similar cases and recommended AV exclusions to mitigate this issue? Such a resource could also be useful for troubleshooting similar problems.

Thanks in advance!

[lukebakken]

recommended AV exclusions to mitigate this issue

Personally I'd recommend removing any and all "security" or antivirus software from Windows servers running RabbitMQ.

Otherwise, create exclusions for RabbitMQ's data in %APPDATA%\RabbitMQ as well as for the installation directory for RabbitMQ as well as Erlang.

[michaelklishin]

@klettier I don't know what kind of "reference page" you are looking for, there is a documentation section.

It is extremely common to see obscure issues on Windwos because of anti-virus software that prevents RabbitMQ from accessing it files, opening network connections, and so on.

[klettier]

I know this may not seem directly related to RMQ, so we don't expect much further discussion on this topic. However, I just wanted to share that whenever this issue occurs, we typically downgrade to RMQ server version 3.12.14. After doing so, the issue no longer persists, which is quite unexpected in this context.

[michaelklishin]

@klettier different versions have different disk I/O and CPU usage patterns for many reasons, that can trigger the anti-virus tool. In any case, there is only one right thing to do here. RabbitMQ 3.12 is out of community support and in a few months will be completely out of support.

[michaelklishin]

Thanks to @tomerrt, we have learned that a filesystem API behavior that can result in difficult to explain eacces errors is not only Windows-specific, it is specific to a certain set of Windows versions, and the newest versions
should not be affected.

While our current workaround helps, it must cover all file deletions, which is a fair number of places even in the CQ storage layer alone. Adopting a newer Windows version can be a viable option that would help those running RabbitMQ 3.13.x or any 4.x version before 4.1.2 where the first slew of workarounds shipped.