Showing entries 61 to 66
« 10 Newer Entries
Displaying posts with tag: Technical (reset)
An underrated cluster admin’s companion: dopd


Split brain, with DRBD, is much less of a disaster than in conventional cluster setups employing shared storage. But, you ask, how can I protect my DRBD cluster against split brain in the first place? Here’s how.

Let’s briefly reiterate what split brain, in the DRBD sense, really means. DRBD split brain occurs when your nodes have lost their replication link due to network failure, and you make both nodes Primary after that.

When just the replication link dies, Heartbeat as the cluster manager will still be able to “see” the peer node via an alternate communication path (which you hopefully have configured, see this post). Thus, there is nothing that would keep Heartbeat from migrating resources to that DRBD-wise disconnected node if it so decides or is so instructed. That would cause precisely the DRBD split …

[Read more]
DRBD 8.2.0 introduces protocol integrity checksums


DRBD 8.2.0, released today, includes a much requested new feature, embodied in the new data-integrity-alg configuration option: DRBD protocol level data integrity checksums.

A few months ago, some users alerted us to DRBD replication issues where DRBD supposedly “ate their data”, i.e. corrupted replicated data in transit. Eventually we traced those problems not to DRBD errors, but in fact to network drivers messing up TCP checksums or segmentation. Typically this was related to using either TCP segmentation offloading (TSO) or TCP checksum offloading. However, at the time DRBD had no way of detecting these errors — you would only find out if you switched over to your Secondary, only to find your data not having been replicated properly.

With DRBD 8.2.0, you can check the integrity of replicated data in transit. To that end, …

[Read more]
Performance tuning DRBD setups


These days, we seem to be getting a lot of inquiries from new or would-be DRBD adopters, especially MySQL and PostgreSQL DBAs wanting to add high availability to their master database servers. And these, unsurprisingly, turn out to be two of their most popular questions:

How will using DRBD affect my write performance?

… and …

What are DRBD’s most important tunables with regard to write performance?

Let’s take a look at both of these issues. Ready? Let’s go.

Basically, there’s usually one important potential bottleneck in any DRBD “Protocol C” (synchronous replication) setup, and it’s not the network connection. With ubiquitous Gigabit Ethernet links available to be dedicated for DRBD replication, network latency and throughput become negligable variables. The …

[Read more]
Why DRBD won?t let you mount the Secondary


As I’m sure you’re aware, DRBD disallows access (any access, including read-only) to a DRBD device in Secondary mode. This always raises questions like the one I’ve taken the liberty to quote here. It came up in a MySQL webinar on replication and HA:

Because of the asynchronous nature of [MySQL] replication we end up with a dilemma when looking at using slaves as read nodes in that the only time we go to the database for information is to build a local cache file, and that local cache file is ONLY removed when information related to that cache file changes, it is NOT based on time. If we had a synchronous method of replication we would then know the cache files were always getting the right information, but because of the asynchronous nature …

[Read more]
Using DRBD directly (without a file system)

Some applications require direct access to a block device, without an intermediate file system. Some Oracle and MySQL configurations are an example, as are some Xen setups, or IET. Can you do this with DRBD? Sure you can.

However, you need to fulfill two prerequisites:

  1. Your application can’t access DRBD while it’s in the Secondary role. So, you must make sure DRBD is Primary before your application attempts using that device.
  2. The user in whose context your application runs needs read/write access to that device.

Your cluster manager, when configured properly, normally takes care of item #1 for you. #2 is a little trickier:
Normally, DRBD’s device nodes are owned by root:disk, with permission bits set to 0660 (rw-rw----). So in order to allow your application to use the device, you have two options:

  • Add …
[Read more]
DRBD and kernel upgrades

A question I recently got from our friends at MySQL:

When speaking about DRBD, we mention that if you upgrade your Linux kernel, it is important that you also upgrade the version of DRBD appropriately.
Question: When upgrading the kernel via yum/yast, does it automatically detect that you should also upgrade your DRBD module?

The answer is, as always, a clear and resounding “it depends.”
Let’s break this down by distribution.

  • If you’re on Debian and updating your linux-image package, dpkg will complain about an unresolved dependency from the DRBD kernel module package, forcing you to update that as well.
  • If you’re on Debian and adding a more recent linux-image package, it’s up to you to remember to install a new DRBD kernel module package as well.
  • If you’re on …
[Read more]
Showing entries 61 to 66
« 10 Newer Entries