Version 0.7 of Q4M (a pluggable message queue storage engine for MySQL) has been released with following changes.

1. Faster SELECT COUNT(*)

   Q4M now caches the number of rows within a table. It is now possible to heavily issue SELECT COUNT(*) queries to monitor queue usage.

2. Dropped binlog capability flags

   Q4M tables were incorrectly marked as binlog-capable in previous releases.

3. Added examples/crawler

   Q4M now includes an example web spider implementation. According to a test using preliminary version (detail in Japanese), the implementation was about two times faster than a crawler …

Today I have uploaded Q4M (a Queue for MySQL) 0.6, which is basically a performance-improvement from previous releases. Instead of using pread's and a small user-level cache, Q4M (in default configuration) now uses mmap for reads with a reader/writer lock to improve concurrency.

I also noticed that it would be possible to consume a queued row in one SQL statement.

SELECT * FROM queue_table WHERE queue_wait('queue_table');

This statement actually does the same thing as,

if (SELECT queue_wait('queue_table') == 1) {
  SELECT * FROM queue_table;
}

But since the former style requires only one SQL statement (compared to two statements of the second one), it has much less overhead.

And combining these optimizations together, consumption speed of Q4M has nearly doubled from previous post (or trippled from 0.5.1) to over 57,000 rows per second. …

Today I had a chance to explain Q4M in detail, and here are the slides I used.

It covers from what (generally) a message queue is, the internals of Q4M, how it should be used as a pluggable storage engine of MySQL, to a couple of usage senarios. I hope you will enjoy reading it.

| View | Upload your own

I always use to blog my temporary ideas on one of my Japanese blog (id:kazuhooku's memos). When I wrote my thoughts on how to further optimize Q4M, Nishida-san asked me "how fast is the raw performance without client overhead?" Although it seems a bit diffcult to answer directly, it is easy to measure the performance of MySQL core and the storage engine interface, and by deducting the overhead, the raw performance of I/O operations in Q4M can be estimated. All the benchmarks were taken on linux 2.6.18 running on two Opteron 2218s.

So at first, I measured the raw performance of MySQL core on my testbed using mysqlslap, which was 115k queries per second.

$ perl -e 'print "select 1;\n" for 1..10000' > …

Today I have released Q4M version 0.3. Changes from the previous are as follows.

Support for message relaying

Messages submitted to Q4M tables can be forwarded to another Q4M table on a different MySQL server. The transfer is trustable, there would be no message loss or duplications even on a server or a network failure. It is also possible to build a message broker above the API used for relaying.

Prioritized subscription to multiple tables

Clients can now subscribe to more than one table at once. The subscription priority can be specified, thus it is able to create a priority queue using multiple tables.

For more information, please refer to the Q4M homepage.

For several years I have been wondering if there was a RDBMS that serves as a message queue as well1. And in the end of last year, noticing that MySQL 5.1 with support for pluggable storage engine was entering its RC stage, and that (from little googling) nobody was creating such storage engine, I dicided to write my own.

Q4M (Queue for MySQL) is a message queue that works as a pluggable storage engine of MySQL 5.1, designed to be robust, fast, flexible. The development started in late December of 2007, and although it is very primitive, operates quite swiftly.
q4m.31tools.com

Yes, it is primitive, and might have stability problems (I do not recommend using Q4M in a production environment), but if you are interested, please have a try.

1: At least Oracle seems to, but I never had a chance to use it.