When you connect a Linux server to a SAN, the network driver will present the available SAN partition as normal SCSI drives. This can be more complicated when more than one Fiber Channel paths are involved. Let’s consider a server having two FC adapters (or a dual port one) and a SAN having two FC channels. For this server, each SAN partitions will appear four times but some FC adapter drivers, like the QLogic ones, will filter them out.

If you are using another brand of Fiber Channel adapters, like Emulex, you will see all of them. I recently worked on a MySQL project using 9 SAN partitions so I ended up dealing with 36 SCSI devices (/dev/sdb to /dev/sdao). Dealing with so many devices is not a fun task, especially if the order of the disks is not preserved between after a reboot. My first attempt to deal with this problem was to use partition label and mount partition by label. Although this solution worked, it was slow and I sometimes …

These days, DRBD is a popular way of achieving HA without having to pay for a SAN and it plays nicely with MySQL. DRBD uses a network connection to mirror two disk partitions on two servers and can use a lot of bandwidth so, usually, the servers are connected through a crossover cable. Recently, I did an engagement where we were setting DRBD for a 100 GB partition on two servers hosted in one of the largest hosting company in the US. For extra reliability, the DRBD private network was using a bond of two Ethernet adapters on each servers.

For a reason that I am not aware of, the hosting company chose to use a VLAN for the private DRBD network instead of a pair of crossover cables. When we started the first DRBD synchronization, loading the private DRBD network at 110 MB/s. At that point, something interesting happened… Here is the explanation of the network engineer of the hosting company. Because of the bonding and the bonding mode chosen …

John Quinn writes about Scaling Drupal he is taking a one step at a time approach and is still writing his 4th and 5 stages.

His first step obviously is separating the drupal from a separate database server, and he chooses mysql for this purpose, moving your DB to a different machine is a good thing to do.

However then he gets this crazy idea of using NFS to share his his drupal shared files :(
(he even dares to mention that the setup ease is good) Folks, we abandonned NFS in the late nineties. NFS is still a recipe for disaster, it has performance issues , it as stability issues (stale locks), and no security admin in his right mind will tolerate portmap to be running in his DMZ.
(Also think about the IO path that …

Morgan is looking in to EC2 on MySQL, so I thought I’d pipe up about stuff I’ve been playing with.

The ephemeral nature of the data is troubling, because at best you’re going to have some lag before you can back stuff up to S3 or some other place. (Unless that was happening continuously… but we’ll come back to that) On the other hand, if you’re doing app sharding or something similar, this essentially just makes you plan that your machines can all die at any time. If you used Google’s semi-sync replication patch, you could easily spin up little replication clusters as needed.

Hm. Clusters. Well, I’m also a fan of MySQL Cluster. What if you ran MySQL Cluster on a single ec2 node (both data and sql nodes)? What if, further, you wrote (and by you, I mean me… code coming soon, I promise) an AsyncFile implementation for Cluster that …

Personally, I’m not a fan of more than one ndbd per machine…

Diamond Notes » Fun with Running a Cluster on Two Servers

Others might argue with this, but I would never put the SQL nodes on the same servers as the ndbd nodes for production. Some say you can run multiple ndbd nodes on the same server and I am more comfortable with that since I can lock the ndbd daemon into memory and know its not going to change (my ndbd nodes on those two servers have been at exactly 71.3% since I started them up. If I had servers for the ndbd nodes that had 16+ gigs of RAM I might start allocating 4 gigs of RAM to a ndbd daemon with 3+ daemons per node. My understanding is that this helps keep the transactional logs for the nodes under control. When you do a ndbd node restart it takes less time for a node to get up and running because of the …

Our friend Dathan recently suggested that The Quality of mySQL lately sucks.

I’ve just ran into yet another obvious bug that has made it’s way into production.

While I’m not going to argue that more unit tests would be a good thing, I’d like to point out that what he’s referring to is not so much a bug as it is a change that happened (and was documented) in 5.0.12:

Beginning with MySQL 5.0.12, natural joins and joins with USING, including outer join variants, are processed according to the SQL:2003 standard. The changes include elimination of redundant output columns for NATURAL joins and joins specified with a USING clause and proper ordering of output columns. The precedence of the comma operator also …

Kris writes:

I had a machine with 4 nics that I wanted to bond 2 by to. I had no problem getting the bond0 device up witn any of the interfaces, however getting a bond1 up always resulted in the above error.

The friendly guys from #centos on freenode pointed me to the missing config.

options bonding mode=4 max_bonds=4

An important thing to keep in mind here is that in the RHEL/CentOS initscripts package, these options are global. There is no way to set a different set of options for each bond. So, if for instance, you had 4 NICs and wanted to have 2 of them bonded in mode 1 and 2 of them in mode 4, you’re SOL. (Unless, of course, you go for insmodding everything by hand. But that’s ugly)

…

It's often quite tempting to make sweeping statements about the superiority of one approach to a problem over another. While various approaches often have advantages, in the real world often there are many competing criteria which make a black and white assessment of choices seem rather simplistic. Politicians and marketing folks oversimplify complex problems every day, and it often makes a real discussion of the issue at hand harder - although it does engender a vitriolic us vs. them approach of yelling. Luckily for us, this never happens in the technical world.

Eric Bergen posted an interesting entry on his blog this morning about DRBD. He makes some very good points, but I believe leaves out some context or assumptions for some of his conclusions, primarily by assuming that there is a single HA setup with a single set of criteria for success …