Very often MySQL is much less stable than we realize. In this video I explain how to detect and dianose these MySQL stalls that last for 5-10 seconds or more.

The reason nobody knows about these issues is that they’re:
a) rare
b) intermittent
c) monitoring software can’t catch them

You can use pt-stalk to detect and diagnose such issues. All of the tools I have mentioned in this video are part of Percona Toolkit (no wonder they all start with PT), you can find the toolkit here:

Percona Toolkit

Here’s the main commands I have discussed in this video:

Starting pt-stalk in foreground

# pt-stalk

Starting pt-stalk in background with email notification:

# pt-stalk …

This post introduces PyMyTools v0.1: my first version of a simple diagnostic toolkit for automating boring DBA tasks. Read on for tool descriptions and demonstrations!

Introduction No lengthy introductions this time:

• I wrote these tools to make my own everyday tasks less time consuming. I hope you too will find them useful.
• Tools can be downloaded from GitHub and are provided under the MIT License.
• Your feedback is very welcome (use GitHub Issues).

Tools: general description The tools are designed to automate some of the simple DBA tasks:

• Monitor the values of server status variables.
• Report on basic status variables (buffer/cache usage, uptime, query traffic).
• Dump …

The details on this issue are here:
https://github.com/facebook/mysql-5.6/issues/369

This test is very simple. I loaded the SSB (star schema benchmark) data for scale factor 20 (12GB raw data), added indexes, and tried to count the rows in the table.

After loading data and creating indexes, the .rocksdb data directory is 17GB in size.

A full table scan "count(*)" query takes less than four minutes, sometimes reading over 1M rows per second, but when scanning the index to accomplish the same count, the database can only scan around 2000 rows per second. The four minute query would take an estimated 1000 minutes, a 250x difference.

I have eliminated the type of CRC32 function (SSE vs non-SSE) by forcing the hardware SSE function by patching the code.

There seem to be problems with any queries …

The details on this issue are here:
https://github.com/facebook/mysql-5.6/issues/369

This test is very simple. I loaded the SSB (star schema benchmark) data for scale factor 20 (12GB raw data), added indexes, and tried to count the rows in the table.

After loading data and creating indexes, the .rocksdb data directory is 17GB in size.

A full table scan "count(*)" query takes less than four minutes, sometimes reading over 1M rows per second, but when scanning the index to accomplish the same count, the database can only scan around 2000 rows per second. The four minute query would take an estimated 1000 minutes, a 250x difference.

I have eliminated the type of CRC32 function (SSE vs non-SSE) by forcing the hardware SSE function by patching the code.

There seem to be problems with any queries …

When your car doesn’t start, you don’t just blindly change the battery, starter, fuel pump, spark plugs or all of the above. Instead, you go to your mechanic and ask him to check what is wrong (or you check it yourself if you are the mechanic) and then fix whatever is broken.

Yet very often I see DBAs doing exactly the opposite with their MySQL servers. Rather than assessing what is the server so busy with, they keep changing configuration options until the problem “goes away”. Alternatively, they add more RAM, more CPUs or faster disks, depending on which resources seems to be the most busy at a time. Or they switch to a new server altogether.

MySQL (with a help of some tools) has a really convenient way to analyse the workload and see clearly what exactly is MySQL so busy doing. And even how much improvement you can expect by, say, fixing a specific MySQL query.

…

PS_history is a tool which collects historical snapshots of the PERFORMANCE_SCHEMA (P_S). This allows you to trend P_S values over time, for example, it is possible to look at the 95 th percentile response time for a query over time.

PS_history is stored procedure and event based, and thus it resides entirely inside of the database with no external dependencies. It uses a clever technique to capture all of the P_S data in one consistent snapshot. This ensures that all of the sys_history views (bundled now with PS_history) have a consistent set of data.

By default, as long as the event_schedule is enabled, PS_history will collect data every 30 seconds. If a snapshot takes 30 seconds, there will be a 30 second delay before the next snapshot starts. This value can be changed by calling the `ps_history`.`set_collection_interval`(N) where N is the number of seconds between samples.

The `sys_history` schema is …

PS_history is a tool which collects historical snapshots of the PERFORMANCE_SCHEMA (P_S). This allows you to trend P_S values over time, for example, it is possible to look at the 95 th percentile response time for a query over time.

PS_history is stored procedure and event based, and thus it resides entirely inside of the database with no external dependencies. It uses a clever technique to capture all of the P_S data in one consistent snapshot. This ensures that all of the sys_history views (bundled now with PS_history) have a consistent set of data.

By default, as long as the event_schedule is enabled, PS_history will collect data every 30 seconds. If a snapshot takes 30 seconds, there will be a 30 second delay before the next snapshot starts. This value can be changed by calling the `ps_history`.`set_collection_interval`(N) where N is the number of seconds between samples.

The `sys_history` schema is …

General query logs and slow query logs remain one of the most popular sources of auditing and diagnostic information in MySQL databases. Customers often ask about the cost of general and slow query logging so I went googling for existing research I could point them to. The data I found was not quite what I hoped for, which is why I decided to do some more testing.

Introduction Logging overhead is an obvious thing and it has been the topic of many blog posts before this one. Sure enough, my Google search returned multiple articles, however I wasn't fully satisfied the findings.
Perhaps the topic is so obvious that DBAs don't revisit it very often. In any case, most of the articles I found were several years old and that presents a few issues:

1. Most of the tests were run on MySQL 5.0 or 5.1.
2. Database hardware isn't what it used to be, which changes the perspective on database performance bottlenecks. …

If there is something that I love about information technology is the incredible amount of unpredictable ways that bugs can bite you where and when you less expect it.

This one I'm about to describe is a serious one, yet there has been a very quiet response from Percona bug team at Launchpad, where I first submitted it (because we use Percona server here). If you have read my other posts you know by now that I am a laz... err, busy guy so bear with me if only today I have verified that this is, in fact, an upstream bug and have raised a bug to Oracle too.
It affects 5.6.33, latest version at the time of this article.

Anyways, the bug started to manifest itself in the form of complete stalls of our main cluster, which was happening at different times and apparently in a way that was unrelated to the amount of traffic hitting the master. When stalling, system CPU time was topping 100% of total available …

FRM-less, transactional data dictionary is arguably the most significant feature change announced MySQL 8.0 development release. The server still has two separate dictionaries (MySQL, InnoDB) but the infamous FRM files are finally being replaced with transactional, InnoDB-based storage.

While this is interesting for various reasons, this particular post will focus on the impact this change has on data dictionary performance.
Test configuration Server configuration:

• Hardware: Amazon EC2 m4.4xlarge, 500GB, 15K IOPS
• OS: Debian Jessie
• MySQL versions tested: 5.7.15, 8.0.0-dmr
• Notable customer MySQL configuration values:
  • innodb_buffer_pool_size = 4GB
  • innodb_log_file_size = 256MB
  • innodb_buffer_pool_dump_at_shutdown = 0
  • innodb_buffer_pool_load_at_startup = 0
  • …