Recently with a client while running a number of disaster recovery tests I came across a nasty situation which was not part of the original plan and provided a far worse disaster situation then expected.
I should preface this condition with some of the environment conditions.
- MySQL 5.0 Enterprise 5.0.54
- RHEL 5 64bit
- Master and 2 Slaves
- MySQL Data and MySQL Binary Logs/MySQL Error Logs are on separate disk partitions
While running stress tests under high load, we tested the filling of partition containing the logs. This partition included the binary log and MySQL error log.
The observed output was.
- An error message was written to the MySQL error log. See below.
- Application throughput dropped, but did not stop.
- Binary logs stopped occuring.
- MySQL proactively stopped logging but continued to process …