Ok, I am perplexed… i don’t say that often.   I have the privilege of testing out a couple of Dolphin Interconnects with Waffle Grid.   They are proving to substantially improve our transactions throughput, I mean we are getting 3x performance over 1gbe…. but what is perplexing is each run going over the faster interconnects results in 1/3 of the memcached sets/gets that occur when testing over 1gbe!  Same datasets, same tests, repeatable results.  See here:

1gbe:
         cmd_get: 771811
cmd_set: 784119

Dolphin:

         cmd_get: 239423
cmd_set: 271259

So instead of testing out the interconnect performance, I am really seeing better results from a higher cache hit rate.  Less items are hitting the LRU.   So what could it …

Lots of news…

We will be giving a Webinar on Waffle Grid on Febuarary 17th , if you want to know more you can find more details  here …  Also as mentioned previously We will also be talking about Waffle Grid at the 2009 MySQL User conference.  So if your interested in learning more about Waffle Grid sign up and stop by.

I am also going to be giving a session at the MySQL User Unconference entitled “Learning from others’ MySQL Performance Mistakes” …  it should be good fun, stop by and say hi.

I …

At the hotel tonight playing around with adding some more stats for Waffle Grid.  I added memcached latency stats to the show innodb status:

———
MEMCACHED
———
Memcached puts    184434
Memcached hits    458
Memcached misses  183976
Memcached Get Total Latency 127 (us)
Memcached Get Recent Latency 127 (us)
Memcached Set Total Latency 6 (us)
Memcached Set Recent Latency 8 (us)

Debating whether or not I want to add disk read stats here as well ( which maybe fun ).  And since the “recent”  is really over the last 1000 calls, do I want to add the last 10 latencies?  something like :

Last 10 gets:  127,129,120,150,400,120,100,120,300,200

Last 10 sets:  9,9,12,15,20,12,10,12,7,8

It has some value in troubleshooting.  Anyone out their have an opinion?  would you …

Some good news.  I have been able to secure some additional hardware for testing over the next few months.  First up I received a pair of Dolphin interconnect cards which I will be putting through the paces sooner rather then later.  These cards will really help complete our Waffle Grid Testing.    Also I had a conversation last week with Texas Memory Systems who has agreed to let me loose on one of their RamSan 500’s  for a few days in February.   And Finally I have reached out and made contact with the folks at FusionIO and Violin Memory to try and setup some time to test with their products as well.  (Still trying to get my hands on one of the new Sun Storage …

By now you have been looking at the UC’s upcoming schedule. On Tuesday @ 2pm Yves and I are currently scheduled to present our presentation on Waffle Grid entitled: “Distributed Innodb Caching with memcached “ . When we submitted this topic, we had not yet come up with a name for the project, So we are really hoping one of the conference gods allows us to change it to something like “The Waffle Grid Project: Distributed Innodb Caching with memcached”. Speaking of changes something, my other accepted proposal a talk on Solid State Disk scheduled for Wednesday @ 2pm Entitled “SAN Performance on a Internal Disk Budget: The coming Solid State Disk revolution” (http://en.oreilly.com/mysql2009/public/schedule/detail/5991) went horribly wrong in the formatting …

As I mentioned earlier their are some limitations with Ec2’s setup and configuration that make it difficult to get Waffle Grid to perform at a high level.  One of the items we are hoping that helps us overcome the Ec2 limitations is Async ( non block ) sets in memcached, unfortunately their current implimentation has some limitations as well, but Brian Aker said they are working on fixing these right now.  But even the current Async implementation showed a slight performance bump on my test hardware, so I decided to give it a spin on Ec2 again.

Thats showing a little more improvement!   43% boost instead of the 22%we saw earlier.  …

So in playing with waffle using memcache non-blocking ( async ) sets I noticed a huge spike in get latency from previous runs on my local hardware, take a look:

Memcached set: Block: 0:112195 : Thread : 1091426640 time:14
Memcached set: Block: 0:58107 : Thread : 1091426640 time:26
Memcached set: Block: 0:58105 : Thread : 1091426640 time:25
Memcached set: Block: 0:58103 : Thread : 1091426640 time:29
Memcached set: Block: 0:58075 : Thread : 1091426640 time:25
Memcached set: Block: 0:58073 : Thread : 1091426640 time:28
Memcached set: Block: 0:58069 : Thread : 1091426640 time:38
Memcached get: Block: 0:112547 : Thread : 1082132816 time:4170
Memcached get: Block: 0:58119 : Thread : 1159866704 time:6065
Memcached get: Block: 0:158858 : Thread : 1084148048 time:6462
Memcached get: Block: 0:125743 : Thread : 1159666000 time:7107
Memcached get: Block: 0:219163 : Thread : …

Cloud computing has been getting lots of buzz over the last few years, and it only seems fair that we talk a little bit about where a Waffle Grid can fit in the cloud. One of our key visions for Waffle Grid is to enable people to add capacity & resources on demand ( as needed ) as load and demand increases without rearchitecting the environment or the application. As I showed previously given the right setup ( disk bound workload, fast interconnect ) Waffle Grid can yield substantial performance benefits. This offers a compelling offering for cloud computing environments.Let’s use a simple example for those that may not fully understand the benefits of cloud computing. Being Christmas time, if you run a ecommerce site that is blessed with increased sales this time of year you probably know that your web infrastructure can sometimes become strained during this time of year. In the past many companies have built their infrastructure to handle …

Since I didn’t have a blog server available in the last few months, Matt offered me to use his BigDBAHead and I accepted.  I must admit, I am not a big blogger but with WaffleGrid, there is a need to communicate more.  Since I have done most of the InnoDB hacking, here is a bit of status and roadmap information.

1. Async memcached_set

Believe it or not… all our results up to now have been done with sync IO…  We were discussing the need of doing async when I discovered memcached_behavior_set and  MEMCACHED_BEHAVIOR_NO_BLOCK. Sometimes, you just feel suddenly tired.  Preliminary results are very interesting, basically, the overhead of the set vanished. We will need to verify no coherence problem are introduced and redo our tests.

2.  Cache coherence between startup

Following a suggestion from Mark Atwood, the memcached key as been …

Yves and I had the rare opportunity to share a gig this week, which means we have been spending a lot of time over meals discussing how to improve Waffle Grid.  Yves will be blogging about some of the things we talked about very soon, I will also be publishing my EC2 experiences with Waffle Grid very soon.  Stay Tuned!