Distributing task with python and zeromq

 

I have a working application using python and zeromq and I would like to optimize it.

Briefly, a master node send the same request to all workers (about 200) and the then collect the answers. Based on the answer, it sends a message back to one node and the node answers back.

Right now I implemented a very simple pattern. Each worker has one REP socket and the server has a list of REQ sockets. The server iterates through all sockets sending the general message and then iterates through all sockets to collect the answers. Finally, based on the answers the server picks one worker, sends a message to it and waits for the reply.

This is, of course, quite slow. The slowest part is sending 200 times the same message. Collecting is also slow. The solutions that I have found to distribute tasks and collect answers do load balance which is not what I need. I need that each worker receives the message and responds.

What is the pattern recommended for this situation?

Thanks

I don't know zmq. Here's a pattern that might not work, just to get started:

a master node send the same request to all workers (about 200)

master PUB bind *:3140 send

worker SUB connect masterhost:3140 SUBSCRIBE recv

the then collect the answers

worker PUSH connect masterhost:3141 send

master PULL bind *:3141 recv

Based on the answer, it sends a message back to one node and the node answers back.

master REQ connect workerhost:3142 send recv

worker REP bind *:3142 recv send

If each worker should need a different job, Pub/sub wont work . What you need then is a worker pool Implementation. Either you push out the jobs in a round robin style (just use the push socket bound on the server and have each client pull from it, zeromq will do the round robin) or you have each worker request a job from the server, if the jobs are at least of some minimum complexity and the variance between jobs is high that is a better approach. There are numerous examples in the zeromq guide on its homepage and on the net:

  • http://zguide.zeromq.org/page:all (look at the paronoid pirate pattern)
  • http://blog.garambrogne.net/post/2010/10/23/simple-python-work-queue-with-zeromq
  • https://github.com/marklit/zeromq-worker-queue

    • The concrete implementation also depends on whether or not you need to have jobs reliably processed.

    链接地址: http://www.djcxy.com/p/18368.html

    上一篇: 如何整合ASP.NET线程模型和ZeroMQ套接字？

    下一篇: 使用python和zeromq分配任务