In this three-part blog series, we introduced a High Availability (HA) Framework for MySQL hosting in Part I, and discussed the details of MySQL semisynchronous replication in Part II. Now in Part III, we review how the framework handles some of the important MySQL failure scenarios and recovers to ensure high availability.

MySQL Failure Scenarios Scenario 1 – Master MySQL Goes Down

• The Corosync and Pacemaker framework detects that the master MySQL is no longer available. Pacemaker demotes the master resource and tries …

In this three-part blog series, we will explain the details and functionality of a High Availability (HA) framework for MySQL hosting using MySQL semisynchronous replication and the Corosync plus Pacemaker stack. In Part I, we’ll walk you through the basics of High Availability, the components of an HA framework, and then introduce you to the HA framework for MySQL.

What is High Availability?

The availability of a computer system is the percentage of time its services are up during a period of time. It’s generally expressed as a series of 9′s. For example, the table below shows availability and the corresponding downtime measured over one year.

Recently I’ve had to rename a node on a production cluster – however, I did not have an experience on this yet so I decided to try on a sandbox and establish a known procedure. Without further ado, assuming we have 3 nodes and we want to rename ha3’s hostname to ha03.

[root@ha01 ~]# crm status
Last updated: Wed Jun  4 21:55:01 2014
Last change: Wed Jun  4 21:53:35 2014 via crm_attribute on ha01.localdomain
Stack: classic openais (with plugin)
Current DC: ha01.localdomain - partition with quorum
Version: 1.1.10-14.el6_5.2-368c726
3 Nodes configured, 3 expected votes
6 Resources configured


Online: [ ha01.localdomain ha02.localdomain ha3.localdomain ]

 Master/Slave Set: ms_MySQL [p_mysql]
     Masters: [ ha01.localdomain ]
     Slaves: [ ha02.localdomain ha3.localdomain ]
 Clone Set: ping_gw_clone [ping_gw]
     Started: [ ha01.localdomain ha02.localdomain ha3.localdomain ]

Here we go:

1. Put ha3 on standby:

   [root@ha01 ~]# …

It is a rather frequent problem to have to manage Virtual IP addresses (VIPs) with a Percona XtraDB Cluster (PXC) or with MySQL master HA (MHA). In order to help solving these problems, I wrote a Pacemaker agent, mysql_monitor that is a simplified version of the mysql_prm agent. The mysql_monitor agent only monitors MySQL and set attributes according to the state of MySQL, the read-only variable, the slave status and/or the output of the clustercheck script for PXC. The agent can operate in 3 modes or cluster types: replication (default), pxc and read-only.

The simplest mode is read-only, only the state of the read_only variable is looked at. If the node has the read_only variable set to OFF, then the writer is set to 1 and reader attributes is set to 1 while if the node has read_only set to ON, the writer attributes will …

Selecting the most appropriate solution for a MySQL HA infrastructure is as much a business and philosophical decision as it is a technical one, but often the choice is made without adequately considering all three perspectives.  When too much attention is paid to one of these aspects at the cost of the others, the resulting system may be over-engineered, poorly-performing, and/or various other flavors of suboptimal.

On Wednesday, June 5, at 10 a.m. PDT (1700 UTC), I will be presenting a webinar entitled, Choosing a MySQL HA Solution, in which we’ll explore the topic of MySQL HA from each of these perspectives.  The goal will be to motivate your thinking about HA in a holistic fashion and help …

Playing around with Percona Replication Manager, and being new to Pacemaker, some errors were somewhat cryptic.

May 26 09:53:39 [2181] ha01.localdomain    pengine:   notice: unpack_rsc_op:    Preventing p_mysql from re-starting on ha02.localdomain: operation monitor failed 'not installed' (rc=5)
May 26 09:53:39 [2181] ha01.localdomain    pengine:   notice: unpack_rsc_op:    Preventing op from re-starting on ha02.localdomain: operation monitor failed 'not installed' (rc=5)

First, it says here that I have ‘operation monitor failed’ on node ha02.localdomain. This error simply means ‘I cannot run the operation monitor because it is not installed’. Where does this operation supposed to come from, from the resource agent. Turns out, I did not have the mysql resource agent in place under /usr/lib/ocf/resource.d/percona.

May 27 23:24:29 [2013] ha01.localdomain crm_resource:    error: unpack_rsc_op:  Preventing reader_vip_2 from …

It never fails. Someone manages to break their Pacemaker cluster, and Henrik starts preaching his usual sermon of why Pacemaker is terrible and why you should never-ever use it. And when that someone is GitHub, which we all know, use and love, then that sermon gets a bit of excess attention. Let's take a quick look at the facts.

read more

It never fails. Someone manages to break their Pacemaker cluster, and Henrik starts preaching his usual sermon of why Pacemaker is terrible and why you should never-ever use it. And when that someone is GitHub, which we all know, use and love, then that sermon gets a bit of excess attention. Let's take a quick look at the facts.

read more

A few thoughts on high availability features (or the current absence thereof) in OpenStack.

I've just proposed a session for the OpenStack Folsom design summit which Jay Pipes was nice enough to invite me to (thanks!), and I thought I'd write up a few thoughts of mine ahead of time to get the discussion started.

read more

The slides from last week’s talks I (co-)presented at Percona Live and Linuxcon Europe are now available from our web site.

• In my tutorial MySQL High Availability Sprint: Launch the Pacemaker, I gave a fast walk-through of Pacemaker high availability for MySQL. Tutorial presented at Percona Live UK 2011 in London, England.
• In Fencing and Maintaining Sanity in High Availability Clusters, Madison Kelly and I explored the importance of fencing in HA clusters, highlighted important considerations for fencing, and shared technical details and best practices. Talk presented at Linuxcon Europe 2011 in Prague, Czech Republic. …