In preparation for an upcoming demo, we decided to use Sphinx for the full text search. We want to do this with Drizzle but hit an obvious problem - Sphinx does not know how to speak to Drizzle to grab documents for indexing. At least that was the case until last night when I created a patch to do so using the new libdrizzle library.

You can download the patch here: sphinx-0.9.8.1-drizzle.patch

I’m releasing this patch BSD, so do with it what you want. I’m hoping Andrew (the Sphinx author) will include this in the main package in an upcoming release. :)

So for now, to get things working, we first need to patch the sphinx-0.9.8.1 source and build/install:

I read Jeremy’s post last week on jobs that matter and was also reminded of Tim O’Reilly’s post about the same topic from earlier this year (which is a great read by the way). This got me thinking, and I realized it has been about one year since I made the commitment to myself to get more involved with open source. I had been a passive member for over ten years, but I decided it was time for a change. After looking at various GNU projects, the Linux kernel, MySQL, and a few others, I decided to focus on the MySQL community. After attending OSCON I was even more inspired with the announcement of Drizzle, along with learning about other projects such as …

Last week at the MySQL Conference & Expo I made a release right before my first Gearman talk. I didn’t get a chance to blog about it, so here it is now. This is a fairly big release since it includes a major refactoring of the job server to now be threaded. Doing some simple tests on it with a 16-core Intel machine:

The Y axis is total jobs/second, and the X axis is number of clients/workers (8×8 means 8 clients and 8 workers). The clients and workers are the ‘blobslap’ utilities included in the source, and they are shoving as many jobs/second through the server as they can. Job size was random between 0 and 1024 bytes. The job server peaked out at about 43,000 jobs/second, most likely due to the machine being so busy (running job server, workers, and clients). The job server uses all non-blocking I/O with a …

Last week at the Drizzle developer day a few people were surprised at the number of plugin interfaces we’ve created already. I created this image to give a sampling of interfaces and plugins:

As you can see, things are quite different from MySQL.

Thanks to everyone who came out for the Gearman and libdrizzle talks at the MySQL Confrence & Expo and the Percona Performance Conference! Links to the slides are provided below:

• The Return of Gearman (Percona)
• libdrizzle: A New Client Library for Drizzle and MySQL (MySQL)
• Map/Reduce and Queues for MySQL Using Gearman (MySQL)

If you want to learn more about Gearman, be sure to check out gearman.org or the upcoming 3-hour tutorial or …

The other day I wrote about what I did inside of Drizzle to create the new Protocol plugin type. A few people have asked me to expand on this a bit more, so here is a diagram of how this will work:

As you can see, you will be able to communicate with the server not just through the traditional client library, but through anything that can implement the Protocol class. The first new class will be the new libdrizzle (which also supports MySQL protocol so Drizzle can talk to those clients), but soon we’ll also be able to handle direct HTTP, Gearman, and possibly memcached requests. This will eliminate the need for the client driver altogether in some applications (for example, an AJAX request could query Drizzle directly and ask for a JSON object to be returned by the module).

You will also be able to do various local or embeddable interfaces through this same interface. …

I’m wrapping up some Gearman work and packing for the users conference, it should be fun and exciting! I thought I’d send out a reminder that Brian and I will be talking about Gearman at both the MySQL UC and Percona Performance conference, including the possibility of a new release of Gearman. Here’s a hint:

lap> ./gearmand/gearmand -vv -t 4
Starting up
Initializing libevent for main thread
Method for libevent: epoll
Trying to listen on 0.0.0.0:4730
Listening on 0.0.0.0:4730
Creating wakeup pipe
Creating 4 threads
Initializing libevent for IO thread
Creating IO thread wakeup pipe
Thread 1 created
[   1] Entering thread event loop
Initializing …

Over the past couple days I’ve been tearing through the network I/O code inside of Drizzle. I moved anything that touched NET into the Protocol class (reading commands, preferences, error checking) and wrapped them with new Protocol methods. I then moved the NET structure out of Session (or THD for the MySQL folks) and into Protocol. At this point all network I/O was going through Protocol, which allowed me to do a proper abstraction layer with the old libdrizzle as a derived implementation class. The next step was to add in a new plugin hook for Protocol, and move the old libdrizzle implementation into a plugin. Since the server is the only thing using the old libdrizzle, all of that code also went into the plugin (note: this means the final death of old libdrizzle!). So now each new connection grabs a Protocol object from the default registered Protocol plugin and Session will use this anytime it …

Good news! If you want to learn more about Gearman, you’ll have plenty of opportunities in the coming months:

• Percona Performance Conference 2009: The Return of Gearman
• MySQL Conference & Expo 2009: Map/Reduce and Queues for MySQL using Gearman
• MySQL Conference & Expo 2009: Gearman at the DotOrg pavillion, come visit our booth!
• OSCON 2009: Gearman: Bringing the Power of Map/Reduce to Everyday Applications (session)
• OSCON 2009: …

I’m pleased to announce three Gearman releases today! They are:

• Gearman C Server and Library - This includes changes to support new part of the protocol, a new Gearman command line tool, and a few other bug fixes and cleanup. Note: This now installs ‘gearmand’ into your preferred ’sbin’ directory, so if you are upgrading be sure to remove the old one from ‘bin’
• Gearman PHP Extension - This is a brand new PHP extension that wraps the C library to provide PHP with a faster client and worker interface. See the README file and the exampes directory for instructions and how to use it.
• Gearman MySQL UDF - New version of …