MySQL 5.7 supports the Address Sanitizer, which checks for several memory related software errors including memory leaks.  It is really nice to see support for the Address Sanitizer built into MySQL.  Unfortunately, when running the mysql tests included in MySQL 5.7.9, the Address Sanitizer reports several memory leaks, which causes some of  the tests to fail.  Here are some of the memory leaks in the MySQL 5.7.9 software found by the Address Sanitizer.
Memory leaks in 'mysqlbinlog'Several of the 'mysqlbinlog' tests fail due to memory leaks in the 'mysqlbinlog' client program.  Here are the details from the ' …

Percona recently open sourced TokuBackup, a library that may be used to take a snapshot of a set of files while these files are being changed by a running application like MySQL or MongoDB.  Making TokuBackup open source allows a larger set of users to improve and extend the software.  This blog describes how the Address Sanitizer and Thread Sanitizer found bugs in the TokuBackup library.

The TokuBackup library is used by Percona Server for MySQL and …

This blog describes how the Address Sanitizer found bugs in the TokuFT (now PerconaFT) storage library.  TokuFT is the storage library used by the TokuDB for MySQL and TokuMX for MongoDB products.  TokuFT is currently tested with valgrind's memcheck tool.  However, memcheck is very slow compared to native execution, so memcheck is not always used for large tests.  This leads to missed bugs.

The Address Sanitizer is a memory …

TokuFT (now called PerconaFT) is the write optimized storage component used by TokuDB for MySQL and TokuMX for MongoDB.  Since TokuFT is its own component, TokuFT can be tested independently of TokuDB and TokuMX.  Some of the TokuFT tests use valgrind's memcheck, helgrind, and DRD tools to identify bugs.

Helgrind and DRD find data races in multi-threaded programs at runtime rather than at compile time.  In my experience with these …

Today’s episode is all about Valgrind – from the pro’s to the con’s, from the why to the how! This episode will be of interest to anyone who is or wants to work with Valgrind on a regular or semi-regular basis.

1. Pro’s/Why
2. Con’s
3. How
   1. Using the latest version
      sudo [yum/apt-get] install valgrind
      #OR#
      sudo [yum/apt-get] remove valgrind
      sudo [yum/apt-get] install bzip2 glibc*
      wget http://valgrind.org/downloads/valgrind-3.10.1.tar.bz2
      tar -xf valgrind-3.10.1.tar.bz2; cd valgrind-3.10.1
      ./configure; make; sudo make install
      valgrind –version # This should now read 3.10.1
   2. VGDB (cd ./mysql-test)
      ./lib/v1/mysql-test-run.pl –start-and-exit –valgrind –valgrind-option=”–leak-check=yes”
      –valgrind-option=”–vgdb=yes” …

Valgrind's helgrind tool identifies data races and lock deadlocks in multi-threaded programs.  Since the MySQL server is a multi-threaded program, can helgrind find any issues in the MySQL server?

A simple test that does nothing except start up the MySQL server and shut it down is run on the MySQL test suite with memcheck and with helgrind.  For this simple test on MySQL 5.7.7, the memcheck run is nice and clean but the …

The ‘uint3korr’ function inside of the mysqld server extracts a 3 byte unsigned integer from a memory buffer. One use is for ‘mediumint’ columns which encode their value in 3 bytes. MySQL 5.6 and MariaDB 10.0 claims to have optimized this function for x86 and x86_64 processors. There is a big comment that says:

Attention: Please, note, uint3korr reads 4 bytes (not 3)!
It means, that you have to provide enough allocated space.

The ‘uint3korr’ optimization may be fast, but it is not valgrind safe. Here is an example where valgrind detects TokuDB reading beyond the end of a buffer when it uses the ‘uint3korr’ function.

==3899== Thread 36:
==3899== Invalid read of size 4
==3899== at 0xB76C089: tokudb_uint3korr(unsigned char const*) (hatoku_defines.h:533)
==3899== by 0xB795C5E: cmp_toku_int(unsigned char*, unsigned char*, bool, unsigned int) (hatoku_cm
==3899== by 0xB797211: compare_toku_field(unsigned char*, unsigned char*, …

A TokuDB MySQL test run with valgrind reported an uninitialized data error when writing into the TokuDB recovery log.

==1032== Syscall param write(buf) points to uninitialised byte(s)
==1032== at 0x3EFA60E4ED: ??? (in /lib64/libpthread-2.12.so)
==1032== by 0xB894038: toku_os_full_write(int, void const*, unsigned long) (file.cc:249)
==1032== by 0xB83248A: write_outbuf_to_logfile(tokulogger*, __toku_lsn*) (logger.cc:513)
==1032== by 0xB83326C: toku_logger_maybe_fsync(tokulogger*, __toku_lsn, int, bool) (logger.cc:836)
==1032== by 0xB8327DE: toku_logger_fsync_if_lsn_not_fsynced(tokulogger*, __toku_lsn) (logger.cc:586)
==1032== by 0xB8493E6: toku_txn_maybe_fsync_log(tokulogger*, __toku_lsn, bool) (txn.cc:600)
==1032== by 0xB7B4EBB: toku_txn_commit(__toku_db_txn*, unsigned int, void (*)(__toku_txn_progress*, void*), void*, bool, bool) (ydb_txn.cc:198)
==1032== by 0xB7B55E9: locked_txn_commit_with_progress(__toku_db_txn*, unsigned int, void (*)(__toku_txn_progress*, …

Last summer my colleague Marko Mäkelä committed this seemingly innocent performance fix for InnoDB in MySQL 5.6:

3581 Marko Makela    2011-08-10
Bug#12835650 VARCHAR maximum length performance impact

row_sel_field_store_in_mysql_format(): Do not pad the unused part of
the buffer reserved for a True VARCHAR column (introduced in 5.0.3).
Add Valgrind instrumentation ensuring that the unused part will be
flagged uninitialized.
Before this, buffers which were used to send VARCHARs from InnoDB to the MySQL server were padded with 0s if the string was shorter than specified by the column. If, e.g., the string "foo" was stored in a VARCHAR(8), InnoDB used to write "3foo00000" to the buffer (the first character - 3 - determines the actual length of the string). However, even though these trailing bytes are not used anywhere, writing 0s to the buffer certainly has a cost. Hence …

Valgrind is a very useful tool. This has been recognized by the mysql developers long ago. They've even invested effort into creating a special mode for the mysql test driver (mysql-test-run.pl) to invoke the server tests under valgrind and collect the results.
But the sad fact remains that this is not exactly usable on MacOSX due to immaturity of the valgrind's support for this platform.
Encouraged by the announcement of the return of MacOSX support in valgrind 3.6 I've started periodically giving a try to the latest valgrind version out of the svn on my 64 bit Snow Leopard to see how the things are progressing. I then report the problems that I see to the valgrind developers.
I just concluded the latest round. I've decided to give it some more effort this time and actually went and …