After some fiddling around with the alternatives, I convinced myself that Gearman is the way to go. Before I proceed to show what I did, though, perhaps it's useful if I spend a few words about Gearman …

The July meeting of the Boston MySQL User Group will feature Eric Day, a prominent Drizzle developer, talking about Drizzle and Gearman:

In this talk we will discuss two growing technologies: Drizzle and Gearman.

We will explain what the Drizzle project is, what we aim to accomplish, and an overview of where we are at. We will also be introducing the fundamentals of how to leverage Gearman, an open-source, distributed job queuing system. Gearman’s generic design allows it to be used as a building block for almost any use - from speeding up your website to building your own Map/Reduce cluster. We will tie Drizzle and Gearman together and demonstrate how they work in a custom Search Engine application.

————————

Here is the URL for MIT’s Map with the location of this building:
…

I just finished adding pluggable protocol support to the Gearman job server, this will enable even more methods of submitting jobs into Gearman. If all the various Gearman APIs, MySQL UDFs, and Drizzle UDFs are not enough, it’s now fairly easy to write a module that takes over the socket I/O and parsing hooks to map any protocol into the job server. As an example module, I added basic HTTP protocol support:

> gearmand -r http &
[1] 29911
> ./examples/reverse_worker > /dev/null &
[2] 29928
> nc localhost 8080
POST /reverse HTTP/1.1
Content-Length: 12

Hello World!

HTTP/1.0 200 OK
X-Gearman-Job-Handle: H:lap:1
Content-Length: 12
Server: Gearman/0.8

!dlroW olleH

I’ve added a few headers for setting things like background, priority, and unique key. For example, if you want to run the above job in the background:

POST /reverse HTTP/1.1
Content-Length: 12
X-Gearman-Background: true

Hello World!

HTTP/1.0 200 …

I’ve been working with Patrick Galbraith for the past couple weeks on a new project that started as an example in his upcoming book. It is a search engine built using Gearman, Sphinx, Drizzle or MySQL, and memcached. Patrick wrote the first implementation in Perl to tie all these pieces together, but there is also a Java version underway bring written by Trond Norbye and Eric Lambert that will be shown at the CommunityOne and JavaOne conferences next week. I’ve been helping get the system setup on a new cluster and with the port to Drizzle.

Narada …

If you missed Gearman or Drizzle at the MySQL Conference, have no fear, a number of folks will be at OSCON too! There will be a many opportunities to learn more or get involved with the two projects:

• 3-Hour Gearman Tutorial - Learn about the latest developments and participate in hands-on demos to help build your own Gearman-powered applications.
• 45-Minute Gearman Session - Get a more concise glimpse about what Gearman is and how to use it.
• 45-Minute Drizzle Panel - Get an update and ask questions about …

First there was LAMP.  But are you using GLAMMP?  You have probably not heard of it because we just coined the term while chatting at work.  You know LAMP (Linux, Apache, MySQL and PHP or Perl and sometimes Python). So, what are the extra letters for?

The G is for Gearman - Gearman is a system to farm out work to other machines, dispatching function calls to machines that are better suited to do work, to do work in parallel, to load balance lots of function calls, or to call functions between languages.

The extra M is for Memcached - memcached is a high-performance, distributed memory object caching system, generic in nature, but intended for use in speeding up dynamic web …

I’m pleased to announce version 0.6 of the Gearman C server and library. The major new feature of this release is a pluggable persistent queue for the job server. It comes bundled with a libdrizzle module (so your queue can live in Drizzle or MySQL), but Brian has already written a libmemcached module and there is a flat-file module in the works as well. The persistent queue allows background jobs to be stored via the pluggable module, so if the job server crashes or is shutdown, the queue module can repopulate the job server with any jobs that were not yet complete. This is just the first version of the queue support, so expect more modules and features in the future!

On a related note, James Luedke has also released version 0.3 of the …

If you’ve pulled the latest Drizzle code from lp:drizzle, you may have noticed a new plugin/gearman_udf directory in there. This is a new UDF that acts as a Gearman client. This is mostly a port of the Gearman MySQL UDF I wrote, but I did it the proper C++ way to fit in better with Drizzle. It also uses the new plugin system Monty Taylor has been working in, which makes it much easier. :)

To use it, just make sure you have the Gearman C library installed and Drizzle will pick it up and build it for you. No extra configuration required!

The following example assumes you have a Gearman job server and a reverse worker running (see examples/reverse_worker in the C library package).

 …

I read Jeremy’s post last week on jobs that matter and was also reminded of Tim O’Reilly’s post about the same topic from earlier this year (which is a great read by the way). This got me thinking, and I realized it has been about one year since I made the commitment to myself to get more involved with open source. I had been a passive member for over ten years, but I decided it was time for a change. After looking at various GNU projects, the Linux kernel, MySQL, and a few others, I decided to focus on the MySQL community. After attending OSCON I was even more inspired with the announcement of Drizzle, along with learning about other projects such as …

Last week at the MySQL Conference & Expo I made a release right before my first Gearman talk. I didn’t get a chance to blog about it, so here it is now. This is a fairly big release since it includes a major refactoring of the job server to now be threaded. Doing some simple tests on it with a 16-core Intel machine:

The Y axis is total jobs/second, and the X axis is number of clients/workers (8×8 means 8 clients and 8 workers). The clients and workers are the ‘blobslap’ utilities included in the source, and they are shoving as many jobs/second through the server as they can. Job size was random between 0 and 1024 bytes. The job server peaked out at about 43,000 jobs/second, most likely due to the machine being so busy (running job server, workers, and clients). The job server uses all non-blocking I/O with a …