I was working on a client’s server that was recently compromised.  Interestingly the attacker had replaced the OpenSSH server with “trojan” binaries that accomplished two things:

• All logins to the system had the username/password logged in a plain-text file (this was how I discovered the problem with OpenSSH)
• It gives the attacker a remote backdoor that is undetected to the system. Another words, when the attacker logs in using the compromised sshd they do not show up in top or ps.

This is bad enough, but it gets worse. The system had been compromised for a week before I began working on it. There is no telling how many other binaries were changed. I did find that the ssl certs had been modded. At this point there is no way that the current setup can be cleaned with any assurance that it is completely secure. No baseline of the system with a tool such as tripwire …

This is actually old news, but I never thought to file a bug report (until now) or say anything to anyone about it. If you use mysqldump to dump and restore a MySQL table that has INSERT triggers, you can get different data in your restored database than you had when you dumped. [...]

This post is for people who are trying to roll back unwanted modifications to their MySQL database.

You cannot use the binary logs to undo unwanted changes to your data. The binary logs are for redoing statements, not undoing them. If you have a backup, you may be able to restore the backup and [...]

The Monolith Toolkit of scripts for DBA routines. 0.4.3 has been released. You can download it here: http://code.google.com/p/monolith-toolkit/

Some information on the toolkit and what it contains:

• mt-backup-parallel -> runs mysql backups in parallel super fast, has lots of reporting features
• mt-check-replication -> script to report on replication status for slave servers
• mt-rhcluster-check-filesystems -> reports on redhat cluster filesystems (for mysql active/passive clustering)
• mt-rhcluster-script-wrapper -> wrapper script for running any of these scripts on a redhat cluster, chooses the active node to run the script on
• mt-connections-log -> logs connections to mysql to disk, reports on threshold overages
• mt-flush-tables-sequence -> runs through schema.tables to flush in …

At the European Customer Conference a couple of weeks back, one of the topics was the use of DRBD. DRBD is a kernel-based block device that replicates the data blocks of a device from one machine to another. The documentation I developed for that and MySQL is available here.

Fundamentally, with DRBD, you set up a physical device, configure DRBD on top of that, and write to the DRBD device. In the background, on the primary, the DRBD device writes the data to the physical disk and replicates those changed blocks to the seconday, which in turn writes the data to it’s physical device. The result is a block level copy of the source data. In an HA solution, which means that you can switch over from your primary host to your secondary host in the event of system failure and be sure pretty certain that the data on the primary and seconday are the same.

…

Updated the release, sure it’s only been a matter of hours but I added the standard cnf files for 2,4,8 and 16GB server installations. Also added the rhcluster-wrapper script that can be used to run just about anything on the active database node when using mysql in an active/passive setup.

Download here:  http://code.google.com/p/monolith-toolkit/

Today I decided to package all of my various scripts together into a useful toolkit. Some are perl, some are shell script. These scripts are, in general, one off scripts that I wrote in order to get things done that weren’t available at the time. Now they’re nicely organized and will receive updates at the google code repo.

So far the toolkit includes the following scripts:

• mt-backup-parallel -> the parallel backup script I wrote about in my last post
• mt-check-replication -> script to report on replication status for slave servers
• mt-check-rhcluster-filesystems -> reports on redhat cluster filesystems (for mysql active/passive clustering)
• mt-connections-log -> logs connections to mysql to disk, reports on threshold overages
• mt-flush-tables-sequence -> runs through schema.tables to …

Large databases, long mysqldump times, long waits for globally locked tables. These problems basically never go away when you rely on mysqldump with –all-databases or a list of databases, as it dumps schemas serially. I’m not going to explain serial vs parallel processing here since that’s a larger topic. Suffice to say that in these days of multi-core / multi-cpu servers we only make use of one processor’s core when we serially export databases using mysqldump. So, I have a new script that attempts to alleviate those issues and now I need testers to provide feedback/improvements.

In order to keep some sanity when dealing with hundreds of database servers, the script takes care of the following:

1. low global locking time requirements: solved by parallel tasks / forked processes
2. backup file checking: with mysqldump files; it checks for “–Dump completed” at the end of the sql file …

To have online backups of MySQL, We recently bought a license for InnoBase/Oracle's InnoDB Hot Backup Tool, ibbackup. This tool, used in conjunction with innobackup, has worked great in creating a nightly backup, with no downtime during the backup. Not even nagios messages!

I run innobackup/ibbackup on one of our slaves (well, it's also a dual master, but not used by apps). innobackup produces a backup in a directory that I specified, and when run results in a time-stamped directory, as show below:


ls -l 2008-09-17_03-00-03/
total 276272
-rw-r--r-- 1 root root 349 2008-09-17 03:00 backup-my.cnf
drwxr-x--- 2 root root 4096 2008-09-17 03:55 grazr
-rw-r--r-- 1 root root 27 2008-09-17 03:55 ibbackup_binlog_info
-rw-r----- 1 root root 186109952 2008-09-17 03:55 ibbackup_logfile
-rw-r----- 1 root root 10485760 2008-09-17 03:00 ibdata1
-rw-r----- 1 root root …

I have completed a rough draft of the first chapter of "Drupal Performance and Scalability". The first chapter of this online book is divided into four sections, the first of which focuses on the importance of fully defining your performance and scalability goals, helping you to identify what you need to accomplish and how to set concrete and attainable goals. The second section discusses monitoring and measuring your ongoing progress, helping you decide what you need to monitor, and how to monitor it. The …