I’ve been following the excellent work that Jan, Kay, and others have been doing with MySQL Proxy, it has really matured into a great piece of software. I talked to Jan at the MySQL UC and toyed with the idea of integrating libdrizzle into MySQL Proxy. I’ve also been asked by a number of folks when a Drizzle Proxy project will be started and if it will be as feature rich as MySQL Proxy. For a while I just said “Someday, I just don’t have the time.” Lately though I am hoping we never have a Drizzle Proxy project.

Let me explain.

One of the fundamental ideas in software engineering is code reuse through libraries or modules. Rather …

I just finished adding pluggable protocol support to the Gearman job server, this will enable even more methods of submitting jobs into Gearman. If all the various Gearman APIs, MySQL UDFs, and Drizzle UDFs are not enough, it’s now fairly easy to write a module that takes over the socket I/O and parsing hooks to map any protocol into the job server. As an example module, I added basic HTTP protocol support:

> gearmand -r http &
[1] 29911
> ./examples/reverse_worker > /dev/null &
[2] 29928
> nc localhost 8080
POST /reverse HTTP/1.1
Content-Length: 12

Hello World!

HTTP/1.0 200 OK
X-Gearman-Job-Handle: H:lap:1
Content-Length: 12
Server: Gearman/0.8

!dlroW olleH

I’ve added a few headers for setting things like background, priority, and unique key. For example, if you want to run the above job in the background:

POST /reverse HTTP/1.1
Content-Length: 12
X-Gearman-Background: true

Hello World!

HTTP/1.0 200 …

We’ve been looking for a Drizzle regression for some time now, and today I decided I would take a step back and make another attempt to find it. The first step in doing this was to reproduce this consistently and find a baseline. We’ve noticed it most dramatically with a 16 concurrent connection test from sysbench in read-only mode. I used two 16-core Intel machines running Linux we have for development. We’ve noticed the regression on certain machines but not all, and these two machines provided one of each. I also setup a MySQL 5.1.35 server to use as a baseline to give some comparisons outside of Drizzle. So first, a few more details on the machines:

Machine 1: 16 core, 16GB RAM, cache sizes from dmesg:
[    0.010000] CPU: L1 I cache: 32K, L1 D cache: 32K
[    0.010000] CPU: L2 cache: 4096K
From /proc/cpuinfo:
cache_alignment : 64

Machine 2: 16 core, 40GB RAM, cache sizes from dmesg:
[ …

I’m pleased to announce a new version of libdrizzle! This is mostly a bug fixing and maintenance release before I start in on more significant development. One of the new features I added was a hook to be able to use your own I/O event mechanism rather than the default poll(). This will allow you to use libraries like libevent, which can be useful when dealing with a large number of file descriptors, or to mix with other file descriptors in your application (for example, you could listen on other fd’s alongside the non-blocking Drizzle/MySQL socket connections). There is not much for examples or documentation yet with this feature, but for now you can email or find me in #drizzle on irc.freenode.net if you would like to know more.

One of the next steps with libdrizzle is a better protocol abstraction, since the Drizzle protocol is diverging quite a bit from how the MySQL …

I’ve been working with Patrick Galbraith for the past couple weeks on a new project that started as an example in his upcoming book. It is a search engine built using Gearman, Sphinx, Drizzle or MySQL, and memcached. Patrick wrote the first implementation in Perl to tie all these pieces together, but there is also a Java version underway bring written by Trond Norbye and Eric Lambert that will be shown at the CommunityOne and JavaOne conferences next week. I’ve been helping get the system setup on a new cluster and with the port to Drizzle.

Narada …

We’ve been looking at high concurrency level issues with Drizzle and MySQL. Jay pointed me to this article on the concurrency issues due to shared cache lines and decided to run some of my own tests. The results were dramatic, and anyone who is writing multi-threaded code needs to be aware of current CPU cache line sizes and how to optimize around them.

I ran my tests on two 16-core Intel machines, one with a 64 byte cache line, and one with 128 byte cache line. First off, how did I find these values?

one:~$ cat /proc/cpuinfo | grep cache_alignment
cache_alignment : 64
...

two:~$ cat /proc/cpuinfo | grep cache_alignment
cache_alignment : 128
...

You will see one line for each CPU. If you are not familiar with /proc/cpuinfo, take a closer look at the full output. It’s a nice quick reference of other things like L2 …

If you missed Gearman or Drizzle at the MySQL Conference, have no fear, a number of folks will be at OSCON too! There will be a many opportunities to learn more or get involved with the two projects:

• 3-Hour Gearman Tutorial - Learn about the latest developments and participate in hands-on demos to help build your own Gearman-powered applications.
• 45-Minute Gearman Session - Get a more concise glimpse about what Gearman is and how to use it.
• 45-Minute Drizzle Panel - Get an update and ask questions about …

Last week I blogged about a patch for Sphinx 0.9.8.1 that allowed Drizzle to be used as a data source (this works much like the MySQL and PostgreSQL options). This past weekend I made a new patch that would work with the latest version, 0.9.9-rc2. It can be found here:

sphinx-0.9.9-rc2-drizzle.patch

The instructions are pretty much the same:

Download sphinx-0.9.9-rc2.tar.gz and sphinx-0.9.9-rc2-drizzle.patch
tar xzf sphinx-0.9.9-rc2.tar.gz
cd sphinx-0.9.9-rc2
patch -p1 < ../sphinx-0.9.9-rc2-drizzle.patch
./buildconf.sh
./configure --with-drizzle
make
make install

See my previous blog post for configuration changes to make Sphinx index from Drizzle. …

I’m pleased to announce version 0.6 of the Gearman C server and library. The major new feature of this release is a pluggable persistent queue for the job server. It comes bundled with a libdrizzle module (so your queue can live in Drizzle or MySQL), but Brian has already written a libmemcached module and there is a flat-file module in the works as well. The persistent queue allows background jobs to be stored via the pluggable module, so if the job server crashes or is shutdown, the queue module can repopulate the job server with any jobs that were not yet complete. This is just the first version of the queue support, so expect more modules and features in the future!

On a related note, James Luedke has also released version 0.3 of the …

If you’ve pulled the latest Drizzle code from lp:drizzle, you may have noticed a new plugin/gearman_udf directory in there. This is a new UDF that acts as a Gearman client. This is mostly a port of the Gearman MySQL UDF I wrote, but I did it the proper C++ way to fit in better with Drizzle. It also uses the new plugin system Monty Taylor has been working in, which makes it much easier. :)

To use it, just make sure you have the Gearman C library installed and Drizzle will pick it up and build it for you. No extra configuration required!

The following example assumes you have a Gearman job server and a reverse worker running (see examples/reverse_worker in the C library package).

 …