Do you test your application systematically when you upgrade or reconfigure your database server? You should! Here’s a real (anonymized) story of what happens if you don’t.

When we upgraded to 5.0.62 (from 5.0.27 and 5.0.45), our code broke for queries like this:

SELECT SUM(amt) FROM daily_amt WHERE day = FROM_UNIXTIME(1222889772);

The problem here was a wrong DATE/DATETIME comparison and other bug fixes in MySQL 5.0.62; it was stricter in enforcing the comparison.

This resulted in an outage and revenue loss to the company.

Daniel and I (mostly Daniel) continue to improve mk-upgrade to make it easy and inexpensive to find these kinds of scenarios before they bite you. Don’t get caught with your pants down — next time you make …

A while ago I wrote about a tool to help make upgrades safer. Since then, we have gotten several people to help sponsor development on this tool, and a few of our customers are using it to help find problems before they upgrade their systems.

I can’t think of a single one of the Maatkit tools that didn’t grow out of the need for deeper insight into some part of the system. This tool is no exception. And as always, these tools are like flashlights. When you crouch down near the floor, and shine your flashlight under the refrigerator, you should expect to find a few things that make you cringe.

The other day, one of our customers was using this tool and we started getting an error. The error was caused by the part of the tool that verifies that result sets are the same. Our thought on how to do this was to checksum the results of a query. You can …

Ryan posted an article on the MySQL Performance Blog about how to use mk-query-digest to analyze and understand your memcached usage with the same techniques you use for MySQL query analysis. This is an idea that came to me during the 2009 MySQL Conference, while talking to our friends from Schooner, who sell a memcached appliance.

It suddenly struck me that the science of memcached performance is basically nonexistent, from the standpoint of developers and architects. Everyone treats it as a magical tool that just performs well and doesn’t need to be analyzed, which is demonstrably and self-evidently false. memcached itself is very fast, true, so it doesn’t usually become a performance bottleneck the way a database server does. But that’s not the point. There is a …

Have you ever wondered how optimized your Memcached installation is? There is a common misconception that one doesn't have to think too deeply about Memcached performance, but that is not true. If your setup is inefficient, you could:

• Burn Memory
• Waste Network Round-Trips
• Store Keys That Never Get Retrieved
• Have a Low Cache Hit Ratio (i.e. query MySQL too much)
• Suffer a fate too horrible to contemplate.

Percona does a lot of consulting around Memcached, so we try to take a quantitative, scientific approach to measuring memcached performance, just like everything else we do.

memcached is basically a key-value in-memory database, so it works well to analyze its traffic with Maatkit's mk-query-digest tool. You can now use …

Suppose that you’re writing a new Maatkit tool (just a random example, really) and its job is to measure the difference in execution of queries. The simplest metric is execution time.

Now suppose that you’re trying to figure out a metric of badness. The query executes in a second on machine 1 and 1000 seconds on machine 2.  That’s a pretty bad change.  How do you quantify this?

Now you’ve got a query that executes in 1ms on machine 1, and 10ms on machine 2.  It’s a tenfold change.  Is it a bad change?  Maybe it’s just the difference in which files were cached in memory, or network latency because someone flooded the TCP pipe and the packets had to be backed off and retried, or something like that.  Is this significant?  How should it contribute to the badness score?

Let’s think of another …

I’ve decided to start writing a book about Maatkit. The working title is Using Maatkit. The goal is to provide another angle onto the tools; the docs are quite good in my opinion, but they only say what the tools do. The book will tell you how to use the toolkit to accomplish tasks.

I have no clue when it’ll be done. There is no schedule. At the current rate, it’ll take a while. I posted an outline to the Maatkit mailing list.

I have a few goals for the project. I want it to be printable-quality material (with an index and professional copyediting, which I have already figured out will cost me a pretty penny), but I also want to be able to update it quickly. I’ve had a number of discussions about this idea over the last year with folks in the …

I’ve had several customers in the last week or so who need a way to verify that their application will work well after an upgrade. I’m seeking input on a new tool to help with MySQL upgrades. Please add comments, either here or on the bug report, or on the mailing list topic.

If someone wants to sponsor this work, that would also be welcomed.

Maatkit version 3519 is ready for download. There are a lot of changes in this release, many of which are incompatible with previous releases. There are also a lot of important new features. Read on for the details.

First, thanks to everyone who contributed to this month’s release. A lot of people have jumped into Maatkit and started committing code. I attribute this to deliberately forcing a more open policy with decisions being made on the mailing list, rather than the former policy of “Percona pays for development, so they have more say than you do” — a snobby and ill-advised way to treat an open-source project. If you are interested in contributing to Maatkit, please ask. Subversion commit rights are being handed out willy-nilly. It’s great!

Here’s a synopsis of this …

The slides for my session on Maatkit at the MySQL Conference are uploaded. They should appear on the speaker presentation files page, but I have heard that it can take a few days. Please let me know if they’re not there by May 1, 2009.

I did not have time to plow through everything in the slides. (I deliberately prepared more material than I could cover, and just stopped when I ran out of time.) Alas, I did not remember to take my voice recorder that day, so I have no audio to share with you!

I’m presenting about Maatkit, the toolkit I created to make life better with MySQL, at the MySQL conference next week.

I’m going to give you a whirlwind tour throught some of Maatkit’s features and functionality. The toolkit is much too large and complex to cover more than a small part of it in depth. So here is your advance warning: I’m going to go through a lot of material, and I won’t be stopping for lengthy discussions :-) The Maatkit documentation is very thorough, and I hope to introduce you to things that could be of use to you, so you can go learn about those topics from the documentation.

Let me give you an idea: when I’m optimizing queries, I open up the output of mk-query-digest in …