How do you do, that voodoo, that Queues Do?

Queues seem to be all over the place right now. Maybe its like when I wanted a VW GTi VR6 a few years back. I kept seeing them pass me on the freeway and thought “crap, everybody is getting this hot new thing and I’m missing out!”.

I think everybody at one point looked at MySQL and tought.. “that would work fine as a queue system”. For low volume stuff, it *is* fine. But then somebody grabs your little transactional, relational, reliable queue system and plugs 5 million messages per hour through it, and somewhere, a man name Heikki cries.

So then you start to look around.. and for those of us who have meager budgets and tend to use open source, there aren’t a lot of choices. The guys at Second Life did some research for all of us…. Once you get through that though, you realize that the needs of second life, a MMORPG, are quite a bit different from your average web app.

So, without further ado, my “queue” system round up.

  • ActiveMQ – This shining star of the queueing world seems to come up quickly in conversation. At Adicio, we actually gave it a good try. The main problem was, we’re a PHP shop. The PHP accessibility comes not through the normal Java Messing Service connector, but “STOMP”.

    Honestly, I’m not a big fan of these giant Apache sponsored java projects. SOLR has changed my mind a bit, as it seems to work well and doesn’t really crash. Then again, I’m not carrying a pager anymore, so maybe it does suck and I’m just not seeing it.

    Anyway, at first, ActiveMQ was winning me over. It was pretty quick.. had a pretty simple setup curve (just start up the latest version, and you have a working persistent queue system), and despite having mountains of documentation that reads like the text spammers shove into their emails randomly to pass bayesian filters, it made sense.

    However, its fall was pretty quick, as the first problem we hit was its Producer Throttling. This probably works fine when you’re using the JMS connector. However, with Stomp, when ActiveMQ decides your queue is too full, and it needs you to stop, it just stops acking your packets. Your stomp client blocks (or spins, in non-block mode) and you wait. This is made worse by the fairly naive php stomp driver, which doesn’t really check to see why its write failed, or even try to see if it can.

    Things got better when that was disabled, but the stomp driver was still haphazard. After figuring out that the Master/Slave protocol requires one to shut down the slave whenever failing back to a downed master, I had had enough. Sionara ActiveMQ.

  • RabbitMQ – This one seems to be a favorite of many. My experience is limited, and I really haven’t tried it that much. Its written in erlang, which I guess automatically makes something “telco reliable”. Cool.
  • QPID – Wow, this one is supposedly INCREDIBLE. “500,000 messages per second per LUN.” . WOW. It also has RedHat’s backing, which is a big win for me.
    In fact, as I write this, I’m doing my best to build and install the latest qpid on CentOS 5.4.

     gcc -DHAVE_CONFIG_H -I. -I. -I./src/config -I./include/ -I/usr/src/redhat/BUILD/xerces-c-src_2_8_0/src -I./src/lexer/ -D_GNU_SOURCE -D_REENTRANT -O2 -g -m64 -mtune=generic -MT mapm_add.lo -MD -MP -MF .deps/mapm_add.Tpo -c src/mapm/mapm_add.c  -fPIC -DPIC -o .libs/mapm_add.o
    ...
    

    In case you’re familiar, I’m there. Oops, thats not qpid. Thats xerces-c. Which I have to build.. and I also have to build xqilla after that. Luckily, 40 other packages required to build qpid were available in the standard CentOS yum repository.

    Another unfortunate reality is that there is no qpid connectivity available for PHP. Unless the php-amqp module works. Its really not clear yet.
    Anyway, this looks like a promising messaging technology. However, this much software leaves a lot of room for things to break.. so, while I will probably complete the build, as I want to find out how it stacks up to the others in terms of simplicity and performance, I think this one is dead.

  • Gearman – Ok I’m going to say it up front. I like this one. Its really not a “queue” system per sé. The name is an anagram of ‘manager’ (say that 5 times fast!). Its one of those great things that came out of the Danga group, the same people who created MogileFS and Memcached.

    Call me stupid, but I like to be able to read things. QPID is in C++, and is so big, I don’t even know where to start. Java gives me the shivers, and I don’t even know what erlang looks like. But damn, who doesn’t like poring over well written C? Thats pretty much what the new C port of gearmand is.

    I’m especially fond of the ease with which one can write a persistence layer. I recently submitted code to make the tokyocabinet queue store better. Its a simple B+Tree store that everybody’s going crazy about these days. Its also written in really nice C.

    The built in ability for gearman clients/workers (producers/consumers) to have a 2 way conversation is especially appealing. Its not like they can just freely pass messages back and forth. But clients can choose to wait for the job they submitted to complete. They can also check on the status of the job fairly easily. Workers can send back two integers (numerator and denominator), which is particularly useful for sending back a count of things done over the count of things to do.

    Combine all this cool stuff with the dead simple ‘gearman’ command line client, and you have a happy Clint. I wrote a little PHP worker that just sits around collecting data sent to it by the other workers running. When it receives a “show_all_workers” message (function in gearman-ese), it just spits back a text report of what it knows. This can be triggered by just saying:

    $ gearman -s -f show_all_workers
    
    Known Workers: 5
    
    dev3.adicio.com_Adicio_App_Reverse_Worker_29336 jobs=26508,restarts=0,memory_MB=1.47,lastcheckin=Thu, 21 Jan 2010 15:33:32 -0800
    dev3.adicio.com_Adicio_App_Reverse_Worker_29333 jobs=19194,restarts=0,memory_MB=1.47,lastcheckin=Thu, 21 Jan 2010 15:33:32 -0800
    dev3.adicio.com_Adicio_App_Reverse_Worker_29356 jobs=29208,restarts=0,memory_MB=1.47,lastcheckin=Thu, 21 Jan 2010 15:33:32 -0800
    dev3.adicio.com_Adicio_App_Reverse_Worker_29370 jobs=27638,restarts=0,memory_MB=1.47,lastcheckin=Thu, 21 Jan 2010 15:33:32 -0800
    dev3.adicio.com_Adicio_App_Reverse_Worker_29332 jobs=10636,restarts=0,memory_MB=1.47,lastcheckin=Thu, 21 Jan 2010 15:33:32 -0800
    
    $
    

    This is pretty damn cool. Now double the fun with MySQL UDF’s, and you have a workable solution for queueing via MySQL trigger.

    So, I can’t help but give this one the nod for simplicity of design. There are no massive books written to explain what gearman does. Just a nice easy C library, and perhaps one of the most important things, a really useful PHP extension.

8 thoughts on “How do you do, that voodoo, that Queues Do?

  1. Actually, you don’t need to shut down ActiveMQ for the master/slave to reconnect. You just need to configure the connection between them using the failover transport and it works like a charm. This feature was added in the ActiveMQ 5.3.0 release. In fact, I just tested this last week for use in a network of brokers in ActiveMQ. The write-up of it is available on my blog:

    http://bit.ly/6loIOu

  2. Bruce,

    I wasn’t very clear in the article above. What I was referring to was the procedure required to recover with a pure Master/Slave.

    http://activemq.apache.org/pure-master-slave.html

    As I said, I’m not looking for the big enterprise solution where I have a SAN or a big NFS appliance that I trust not to die (a really nice Linux or FreeBSD server running nfs doesn’t count).

    With pure master/slave, when the master fails the slave takes over and conceivably your stomp clients could be made to use it. However, when the master comes back online, according to the link above:

    “# A failed master cannot be re-introduced without shutting down the the slave broker (no automatic failback)
    # There is no automatic synchronization between brokers. This is a manual process.”

    And the procedure:

    “This is a manual process – once a master has failed, the only sure way to ensure that the toplogy is synchronized again is manually:

    * shutdown the slave broker (The clients do not need to be shutdown – they will wait until the topology is re-established if they are failover clients
    * copy the data directory from the slave over the data directory of the master broker
    * re-start the master and the slave

    Stomp doesn’t really have the “failover” bit builtin, though it can be made to do it more or less.

    My issue with this is that one of the big points of using a queueing system is to be able to quickly dump messages into it and forget about them, trusting that a backend process will receive it.

    But if I have to wait while a big backed up queue is synchronized, even if its just for 1 minute.. thats 1 minute of browser spinning, or errors due to timeouts.

  3. Ah, yes, that’s a very different scenario and is known by a few names including infinite failover and continuous availability. This feature is not available in ActiveMQ largely because it’s a very difficult problem to solve for many reasons. Some of the commercial MOMs have solved this and that’s partially why they cost so much money. But they also come with some very costly requirements to make such a feature work, too. Thanks for the clarification.

    Bruce

  4. I admire the useful data you offer you within your articles. I will bookmark your weblog and have my young children check up here normally. I am fairly confident they’ll discover a lot of new stuff here than anybody else!

  5. Hi, What is the best free software to automatically backup wordpress database and files ? A software that is trustworthy and would not hack your password in wordpress. Have you tried it ? For How long ? Thanks……

Leave a Reply

Your email address will not be published. Required fields are marked *

*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>