Showing entries 1 to 2
Displaying posts with tag: PERC (reset)
The blog was down yesterday

The brief outage was due to a scheduled move of the servers to a separate rack and subnet dedicated to our work with the Center for Information Assurance & Cybersecurity (ciac) at the University of Washington Bothell (uwb), and a11y.com

I am currently exercising the new (to us) equipment and hope to winnow the less than awesome equipment over the next quarter. I spent the last six months finding the best in breed of the surplussed DL385 and DL380 chassis we (work) were going to have recycled. The team and I were able to find enough equipment to bring up one of each with eight and six gigs of memory, respectively. These will make excellent hypervisors for provisioning embedded instances of Slackware, Fedora, RHEL, CentOS, Debian, FreeBSD, OpenSolaris, OpenIndiana, FreeDOS, etc.

When I initially configured this xen paravirt environment, I failed to plan for integration with libvirt, so I am now re-jiggering the software bridges so …

[Read more]
The perils of uniform hardware and RAID auto-learn cycles

Last night a customer had an emergency in selected machines on a large cluster of quite uniform database servers. Some of the servers were slowing down in a very puzzling way over a short time span (a couple of hours). Queries were taking multiple seconds to execute instead of being practically instantaneous. But nothing seemed to have changed. No increase in traffic, no code changes, nothing. Some of the servers got sick and then well again; others that seemed to be identical were still sick; others hadn’t shown any symptoms yet. The servers were in master-master pairs, and the reads were being served entirely from the passive machine in the pair. The servers that were having trouble were the active ones, those accepting writes.

The customer had graphs of system metrics (++customer), and the symptoms we observed were that Open_tables went sharply up, the buffer pool got more dirty pages, InnoDB started creating more pages, and the …

[Read more]
Showing entries 1 to 2