Short answer:
At the publisher, we must be very careful about HWM, because there are many reasons for the failure (from memory) that affect the system as a whole (because the publisher serves all subscribers).
There are also cases in the subscriber when the regulation of HWM can be useful, but this depends mainly on the nature of the subscriber, on what he does with the received message, and on the likelihood that he will not be able to process a large number of received messages in a timely manner; and the expected runtime (how much memory is available, the number of subscribers, etc.).
More detailed answer:
ZMQ uses the HWM concept (high water mark) to determine the capacity of inner pipes. Each connection from the socket or to the socket has its own channel and HWM for sending and / or receiving, depending on the type of socket. Some sockets ( PUB , PUSH ) have only send buffers . Some ( SUB , PULL , REQ , REP ) have only receive buffers . Some ( DEALER , ROUTER , PAIR ) have both send and receive buffers.
Available option :
ZMQ_SNDHWM : set the high water mark for outgoing messages (... to the publisher socket )ZMQ_RCVHWM : set the high water mark for incoming messages (... to the subscriber socket )
ZMQ 3.0+ creates default limits for internal buffers (called HWMs), since HWMs are a great way to reduce memory overflow problems.
Both the ZMQ_PUB and ZMQ_SUB parameters have the ZMQ_HWM parameter set to "Drop", so when the limits are ZMQ_HWM , the memory of the subscriber or publisher should stop growing, at least depending on the ZMQ buffers.
Typically, those who need more protection against illegible memory usage (due to memory issues), publishers:
For inproc transport, the sender and receiver use the same buffers, so the real HWM is the sum of the HWMs set by both parties.
But if you use TCP, and the subscriber is slow, messages will be queued at the publisher.
Common causes of PUB-SUB failure include:
- Subscribers may receive messages too slowly, so the queues grow and then overflow.
- Networks can become too slow, so publisher and publisher line overflows fail.
The message queue in the publisher causes the publishers to run out of memory and crashes, especially if there are many subscribers, and for performance reasons it is not possible to put them on disk.
From the publisher’s point of view, the big strategy that we use with the correct ZWM settings is: Stop queuing new messages after a while , new messages are simply rejected or discarded; this is what ØMQ does when the publisher installs HWM.
ZMQ can also send queue messages to the subscriber
If someone runs out of memory and crashes, it will be more a subscriber than a publisher, which is true. This is ideal for "peak" streams in which the subscriber may keep up with some time, but can catch up when the stream slows down.
Note: HWMs are not accurate; while you can receive up to 1000 messages by default, the size of the real buffer can be much lower (only half) due to the way libzmq implements its queues.
The main source of this assumption is the Pieter Hintjens book, “Attached Volume 1 Code,” available electronically in electronic format; it has a chapter on the High Water Signs containing more detailed explanations on the subject.