Overview The Skinny

Part of the power of Tungsten Clustering for MySQL / MariaDB is the ability to perform true zero-downtime maintenance, allowing client applications full access to the database layer, while taking out individual nodes for maintenance and upgrades. In this blog post we cover various types of maintenance scenarios, the best practices associated with each type of action, and the key steps to ensure the highest availability.

Important Questions Understand the Environment as a Whole First

There are a number of questions to ask when planning cluster maintenance that are critical to understand before starting.

For example:

1. What is the cluster topology?
   • Standalone (connectors write to single cluster master)
     Single cluster: …

The Question Recently, a customer asked us:

How would we manually move the relay role from a failing node to a slave in a Composite Tungsten Cluster passive site?

The Answer The Long and the Short of It

There are two ways to handle this procedure manually when the usual switch command fails to work as expected. One is short and reasonably automated, and the other is much more detailed and manual.

Of course, the usual procedure is to just issue the switch command in the passive cluster:

use west
set policy maintenance
switch
set policy automatic

The below article describes what to do when the switch command does not move the relay role to another node.

SHORT

Below is the list of cctrl commands that would be run for the basic, short version, which (aside from handling policy changes) is really only three …

The Question Recently, a customer asked us:

Where are the logs for a Tungsten Cluster and which are the proper log files to monitor if I do a master role switch to another node?

The Answer: Part I Both basic and advanced logs are available!

The basic logs are symbolic links into the log subdirectory for each layer of the cluster.

For example, this is the listing of the default log directory, /opt/continuent/service_logs:

connector-user.log -> /opt/continuent/tungsten/tungsten-connector/log/connector-user.log
manager-user.log -> /opt/continuent/tungsten/tungsten-manager/log/manager-user.log
mysqldump.log -> /opt/continuent/tungsten/tungsten-replicator/log/mysqldump.log
replicator-user.log -> /opt/continuent/tungsten/tungsten-replicator/log/replicator-user.log
xtrabackup.log -> /opt/continuent/tungsten/tungsten-replicator/log/xtrabackup.log

As you can see, each log file is a …

The Question Recently, a customer asked us:

What would cause a node switch to fail in a Tungsten Cluster?

For example, we saw the following during a recent session where a switch failed:

cctrl> switch to db3 

SELECTED SLAVE: db3@alpha 
SET POLICY: MAINTENANCE => MAINTENANCE 
PURGE REMAINING ACTIVE SESSIONS ON CURRENT MASTER 'db1@alpha' 
PURGED A TOTAL OF 0 ACTIVE SESSIONS ON MASTER 'db1@alpha' 
FLUSH TRANSACTIONS ON CURRENT MASTER 'db1@alpha' 
Exception encountered during SWITCH. 
Failed while setting the replicator 'db1' role to 'slave' 
ClusterManagerException: Exception while executing command 'replicatorStatus' on manager 'db1' 
Exception=Failed to execute '/alpha/db1/manager/ClusterManagementHelper/replicatorStatus alpha db3' 
Reason= 
CLUSTER_MEMBER(true) 
STATUS(FAIL) 
+----------------------------------------------------------------------------+ 
|alpha | …