How do I know that I have consumed all the data?

With a high volume of data there is always a possibility that some of data might not be consumed and it is important to know how you can find out that information and take action based on the result.

The system has a few restrictions in place and you will not receive messages that have a Time-To-Live (TTL) greater than 30 seconds or there is a backlog of 90 000 messages. When at least one restriction is met, the system will considered expired the oldest messages and these will not be available for consumption anymore.

In order to know if there are messages available for consumption you need to use the queue.declare method with the passive option set as True. By doing that you will receive the queue depth, the number of messages available for consumption, and the number of consumers.

If the number of messages is always greater than zero that means that these are available for consumption. One possible solution is to increase the number of consumers, so messages will be consumed faster and the queue depth will go to zero. Just remember that you cannot have more than 10 consumers per queue.

Please don’t hesitate to get in touch with us via email at helpdesk@cirium.com.