Ok, I am perplexed… i don’t say that often.   I have the privilege of testing out a couple of Dolphin Interconnects with Waffle Grid.   They are proving to substantially improve our transactions throughput, I mean we are getting 3x performance over 1gbe…. but what is perplexing is each run going over the faster interconnects results in 1/3 of the memcached sets/gets that occur when testing over 1gbe!  Same datasets, same tests, repeatable results.  See here:

1gbe:
         cmd_get: 771811
cmd_set: 784119

Dolphin:

         cmd_get: 239423
cmd_set: 271259

So instead of testing out the interconnect performance, I am really seeing better results from a higher cache hit rate.  Less items are hitting the LRU.   So what could it …

One of our customers reported strange problem with MySQL having extremely poor performance when sync-binlog=1 is enabled, even though the system with RAID and BBU were expected to have much better performance.

The problem could be repeated with SysBench as follows:

PLAIN TEXT CODE:

1. ./sysbench --num-threads=2 --test=oltp --oltp-test-mode=complex --oltp-table-size=100000 --oltp-distinct-ranges=0 --oltp-order-ranges=0 --oltp-sum-ranges=0 --oltp-simple-ranges=0 --oltp-point-selects=0 --oltp-range-size=0 --mysql-table-engine=innodb  --mysql-user=root --max-requests=0 --max-time=60 --mysql-db=test run

On Dell R900 with CentOS 5.2 and ext3 filesystem we get 1060 transactions/sec with single thread and sync_binlog=1 while with 2 threads …

Some good news.  I have been able to secure some additional hardware for testing over the next few months.  First up I received a pair of Dolphin interconnect cards which I will be putting through the paces sooner rather then later.  These cards will really help complete our Waffle Grid Testing.    Also I had a conversation last week with Texas Memory Systems who has agreed to let me loose on one of their RamSan 500’s  for a few days in February.   And Finally I have reached out and made contact with the folks at FusionIO and Violin Memory to try and setup some time to test with their products as well.  (Still trying to get my hands on one of the new Sun Storage …

One typical question which frequently pops up is whenever it is better to use hardware upgrade or optimize software more. I already wrote about it, for example here.

Today I'll look at the same topic from the consultants view. When consultant should suggest hardware upgrade and when it is not in a simple checklist form.

How good is hardware ? Sometimes people use so crappy hardware it would be much cheaper for them to upgrade before purchasing much of professional services. Though in some cases people like their system to be optimal and so they want to run it on some old box even if it costs them more to optimize it. It may be valid choice allowing to take a hardware boost later down the road when you need a major performance increase and do not …

As one of the co-founders of the Waffle Grid project, I beam with pride every time I get a stellar benchmark or every time I find a new use for the Waffle.  But as a professional I still have to be critical of all solutions I would recommend or deploy.  One of the big goals of Waffle Grid is to replace disk IO which should be slow with remote memory which should be much faster.  But what happens when the disk is no longer slow?  This leads me to ask myself, is Waffle Grid only good for servers with slower disk?  Or is this a solution that can also help systems with fast disk?  So which should you deploy SSD -vs- Waffle?  Are they competitors?  Or are they complementary technologies?

I am going to say this, in these tests latency is king.  The faster the drives can deliver data, the higher the benchmarks should be.  Basically if my interconnect can deliver faster then the drive can serve up …

Following up on my previous post Peter asked if he could see a 24 hour run on the Intel drive…  took me a few days because I am doing some testing on a few other things, but I kicked a run off yesterday before leaving the Vegas airport…  Here ya go:

These are roughly 10 minute TPM averages.  As you can see their is a definate decline in performance the longer the drive is active.   Its realtively small% wise, only about 6% off of peak…  but the delcine is easy to spot.

What’s this mean?  It you beat the hell out of the drive over a long contious period things slow down the longer the drive is in continuous use.   Just an FYI, the drive was 78% full during these tests.

As you may have recently seen there are some articles about scaling MySQL one 256-way system.

I though wow did they really make it work, considering how many bottlenecks remain in MySQL.

What article really tells us ?

First the number 256 - this is not number of Cores... this is number of hardware threads which is not exactly the same thing. Each T2 Plus CPU has 8 cores, which with 8 threads each giving 64 threads per chip or 256 threads all together.

Now what is about MySQL scaling to use these 32 cores with 256 threads ? Especially with the goal of "Do it with minimal tuning i.e as close as possible as out-of-the-box" ? Do we simply start MySQL server and change couple of defaults to make it work ?

No! To get really …

Thought I would throw this out here so I remember it, and it maybe useful for someone else.  During a session here at Sun’s CEC conference one of the presenter was talking about setting CPU affinity for hardware interrupts.  This made me think of our hack ( workaround ) for innodb scalability on a 16 core machine ( set the mysqld affinity to cpu’s 0-7 or 8-15 ).   In order to get the top performance out of a 16 core machine we probably want to ensure everything else goes to the cpu’s not set to be servicing mysql.   So a quick google search came up with this article on setting the affinity for IRQ’s in the linux kernel.   So with a 16 core box, I could set the affinity for Mysqld to cpu’s 8-15, while setting the affinity of the interrupts to something in the 0-7 range.

Did you see the press release today, called “Sun MySQL Server Solution“?

What it means is that we start seeing the fruits of what we’ve been talking about all along since the acquisition, such as when launching MySQL this week in Argentina and in Uruguay:

• That Sun remains committed to MySQL’s cross-platform presence, doing at least as much as before the acquisition on any individual platform or for any individual stack.

• That the acquisition of MySQL through …

As promised, here are the DBT2 results for the Intel SSD drive:

As you see the Intel drive blew away all the competition here… even besting another dbt2 score I got from a nice new shiny 8 disk raid 10 system.