RabbitMQ is a messaging broker installed with Coveo Enterprise Search. It runs as a Windows service along with other Coveo services. Coveo Search Provider adds item into RabbitMQ queues (one per index) and the messages get picked up by the Queue Crawler, which runs as part of the Coveo Enterprise Search service.
From time to time, RabbitMQ goes rabid 😉 and stacks all the messages in the queues causing one or more queue(s) to hold a lot of messages. I have seen up to 2 million stacked messages at one time for the web index queue. This results in physical files which take up all of the disk space, effectively pushing CES service to make all your indexes read-only.
Therefore, it is a good practice to have the Coveo and RabbitMQ installed on a non-OS drive.
Here is the link to the article from Coveo: RabbitMQ Consumes a Large Amount of Disk Space
There are multiple options to help limit this behavior, for instance, you can add an alert in the Coveo Admin Console to notify you when an error occurs.
First, open the Coveo Admin and click on the Configuration link:
Once you are in the configuration section, click on Alerts link:
Configure the email settings and click Accept Changes link.
You can also limit the Message limit on RabbitMQ. This will stop the queue from overloading and would also prevent the server from taking up all of the disk space.
Erlang is a general-purpose, concurrent, garbage-collected programming language and runtime system used by RabbitMQ.
RabbitMQ and Erlang are a third-party solutions and you cannot expect Coveo to support it 100%. I do not take responsibility for the following settings/changes to RabbitMQ. These changes worked for me but I cannot guarantee that they will work for you.
First you would need to set the location for the Erlang’s ERTS (Erlang Run-Time System Application) folder. To do this go into your computer properties > Advanced System Settings > Environment Variables.
Add a system variable by clicking he New button:
Add a variable with name ERLANG_HOME and the value pointing to the directory of the ERTS install folder which is usually DRIVELETTER:\Program Files\Coveo Enterprise Search 7\Third-Party\erl5.10.3\erts-
Click OK and confirm again by clicking OK on the Environment Variables dialog.
Next, we need to find and copy the .erlang.cookie file. This file can usually be found in C:\Windows\.erlang.cookie path for a standard RabbitMQ install. For Coveo RabbitMQ install you can find the file in DRIVELETTER:\Program Files\Coveo Enterprise Search 7\Third-Party folder.
Copy the .erlang.cookie file from that path to C:\Users\%USERNAME%\.erlang.cookie
You can find the destination folder by running %HOMEDRIVE%%HOMEPATH% in the Run command.
The file copy will grant you access to Add/Change policy on RabbitMQ. You can either change the policy by running the following query:
rabbitmqctl set_policy NAMEOFYOURPOLICY "^THEWEBINDEXQUEUENAME$" "{""max-length"":NUMBEROFMESSAGES}" --apply-to queues -n NODENAME
Parameters and the values:
NAMEOFYOURPOLICY: provide a useful name like Coveo Web
THEWEBINDEXQUEUENAME: this is the name of the queue representing your web index
NUMBEROFMESSAGES: this is the max number of message you want to cap it at. A number like 500000.
NODENAME: This is the node name of the install. For a typical Coveo RabbitMQ install the name is not the same as the default RabbitMQ install. Look on the Overview tab for the node name.
Or you can do it via the admin console for RabbitMQ which can be found at http://localhost:15672 on the machine with CES installed.
Once you add the policy, it will take effect immediately. Please keep in mind that by adding the Max Length value, the Messages will be dropped or dead-lettered from the front of the queue to make room for new messages once the limit is reached.
Please let me know if you have any questions or comments.