Publish/Subscribe reliable messaging: Redis VS RabbitMQ

 

Background

I am making a publish/subscribe typical application where a publisher sends messages to a consumer.

The publisher and the consumer are on different machines and the connection between them can break occasionally.

Objective

The goal here is to make sure that no matter what happens to the connection, or to the machines themselves, a message sent by a publisher is always received by the consumer .

Ordering of messages is not a must.

Problem

According to my research, RabbitMQ is the right choice for this scenario:

  • Redis Vs RabbitMQ as a data broker/messaging system in between Logstash and elasticsearch

    • However, although RabbitMQ has a tutorial about publish and subscriber this tutorial does not present us to persistent queues nor does it mention confirms which I believe are the key to making sure messages are delivered.

    On the other hand, Redis is also capable of doing this:

  • http://abhinavsingh.com/customizing-redis-pubsub-for-message-persistence-part-2/

    • but I couldn't find any official tutorials or examples and my current understatement leads to me to believe that persistent queues and message confirms must be done by us, as Redis is mainly an in memory-datastore instead of a message broker like RabbitMQ.

    Questions

  • For this use case, which solution would be the easiest to implement? (Redis solution or RabbitMQ solution?)
  • Please provide a link to an example with what you think would be best!

    • Background

    I originally wanted publish and subscribe with message and queue persistence.

    This in theory, does not exactly fit publish and subscribe:

  • this pattern doesn't care if the messages are received or not. The publisher simply fans out messages and if there are any subscribers listening, good, otherwise it doesn't care.

    • Indeed, looking at my needs I would need more of a Work Queue pattern, or even an RPC pattern.

    Analysis

    People say both should be easy, but that really is subjective.

    RabbitMQ has a better official documentation overall with clear examples in most languages, while Redis information is mainly in third party blogs and in sparse github repos - which makes it considerably harder to find.

    As for the examples, RabbitMQ has two examples that clearly answer my questions:

  • Work queues
  • RPC example

    • By mixing the two I was able to have a publisher send to several consumers reliable messages - even if one of them fails. Messages are not lost, nor forgotten.

    Downfall of rabbitMQ:

  • The greatest problem of this approach is that if a consumer/worker crashes, you need to define the logic yourself to make sure that tasks are not lost. This happens because once a task is completed, following the RPC pattern with durable queues from Work Queues, the server will keep sending messages to the worker until it comes back up again. But the worker doesn't know if it already read the reply from the server or not, so it will take several ACK from the server. To fix this, each worker message needs to have an ID, that you save to the disk (in case of failure) or the requests must be idempotent.
  • Another issue is that if the connection is lost, the clients blow up with errors as they cannot connect. This is also something you must prepare in advance.

    • As for redis, it has a good example of durable queues in this blog:

  • https://danielkokott.wordpress.com/2015/02/14/redis-reliable-queue-pattern/

    • Which follows the official recommendation. You can check the github repo for more info.

    Downfall of redis:

  • As with rabbitmq, you also need to handle worker crashes yourself, otherwise tasks in progress will be lost.
  • You have to do polling. Each consumer needs to ask the producer if there are any news, every X seconds.

    • This is, in my opinion, a worst rabbitmq.

    Conclusion

    I ending up going with rabbitmq for the following reasons:

  • More robust official online documentation, with examples.
  • No need for consumers to poll the producer.
  • Error handling is just as simple as in redis.

    • With this in mind, for this specific case, I am confident in saying that redis is a worst rabbitmq in this scenario.

    Hope it helps.

    Regarding implementation, they should both be easy - they both have libraries in various languages, check here for redis and here for rabbitmq. I'll just be honest here: I don't use javascript so I don't know how are the respected libraries implemented or supported.

    Regarding what you didn't find in the tutorial (or maybe missed in the second one where there are a few words about durable queues and persistent messages and acknowledging messages) there are some nicely explained things:

  • about persistence
  • about confirms (same link as you've provided in the question, just listing it here for clarity)
  • about reliability

    • Publisher confirms are indeed not in the tutorial but there is an example on github in amqp.node's repo.

    With rabbit mq message travels (in most cases) like this
    publisher -> exchange -> queue -> consumer
    and on each of these stops there is some sort of persistence to be achieved. Also if you get to clusters and queue mirroring you'll achieve even greater reliability (and availability of course).

    i think they are both easy to use as there are many libraries developed for them both.

    There are a fews to name such as disque, bull, kue, amqplib, etc...

    The documentations for them are pretty good. You can simple copy and paste and have it running in few mins.

    I use seneca and seneca amqp is a pretty good example

    https://github.com/senecajs/seneca-amqp-transport

    链接地址: http://www.djcxy.com/p/34162.html

    上一篇: 通过WCF提供RabbitMQ服务器对

    下一篇: 发布/订阅可靠消息：Redis VS RabbitMQ