If you use replication with MySQL and memcache at the same time you have the problem to make sure that the memcached and the slave are in sync. If you announce the memcached to mark a entry as dirty and let it update the value from the slave BEFORE it is updated, you fetch a old value.

The easiest way to ensure that memcache is always notified after the slave is updated, you can use ... well ... replication. MySQL 5.1 brings Row Based Replication that makes this kind of applications a lot easier.

I'm not there to present you a Proof of Concept, I'm only close enough to explain the idea:

• Slave fetches records from Master
• Slave applies the records
• Slave exposes its changes as replication stream to the MySQL Proxy
• MySQL Proxy decodes the RBR records, extracts the Primary Key and updates the content in the memcache server

The MySQL Proxy acts as replication client …

The main focus of version 0.2 was to improve handling of almost all of SQL. So now you can issue DESC TABLE tbl_name or RENAME TABLE tbl_name TO another_tbl_name on a partitioned table and you get correct results for SHOW TABLES etc.
Other statements are rejected and we settled down for the feature set we want to provide for full partition scans.
In addition to that there are some performance improvements.

See the full list of changes.

For the next release (0.3) we focus on the dictionary based partition lookup module and further performance improvements.

Finally, there is a new project home page: http://www.hscale.org.

Earlier today I posted these benchmark results testing HSCALE and MySQL Proxy performance.

As Jan Kneschke (the author of MySQL Proxy) pointed out there are quite some improvements in the current development version (svn trunk). So I gave revision 369 a try.

Tests were all the same as mentioned in my previous post. And indeed we see quite dramatic improvements. While the performance of the Lua script stayed almost the same the footprint of the proxy itself sank to only 50 to 65%. Here are the numbers:

As part of developing HSCALE, a partitioning / sharding solution, I set up a benchmark test suite. I made it scripted and thus repeatable to monitor the progress and performance regressions during the development.

Test Suite

The test suite uses mysqlslap to benchmark the overhead of MySQL Proxy itself in real life scenario as well as the different components of HSCALE – query analyzing and query rewriting. The complete test suite is available in the svn trunk at http://svn.hscale.org under hscale/test/performance/mysqlslap. There you find a build.xml – an Ant buildfile that is used to set up the …

This is an idea I came across at the MySQL UC last week: How many applications handle failing COMMITs correctly ? And how can we test it ?

COMMITs can fail. The most simple case are deadlocks. The sad side of deadlocks is that they only happen under real load when you application creates concurrency against the same rows. That's usually hard to create in test-setups.

With MySQL Proxy you can create deadlocks easily. Well, more or less.

You can at least fake them nicely and let the application and server think that we have a deadlock. The trick is:

To make it a bit more interesting you don't have to let all the COMMITs fail. Just let 50% of them fail.

--[[

Let a random number of transaction rollback

As part of the QA we have to verify that applications handle
rollbacks nicely. To simulate a deadlock we turn client-side COMMITs
into server-side ROLLBACKs and return a ERROR packet to the client
telling it a …

If you use a transactional storage engine in MySQL like InnoDB some of your transaction may be terminated by the storage engine because of deadlocks. Sadly it is a bit tricky to see what has led to the deadlock. SHOW ENGINE INNODB STATUS gives only a very minimal look into the state when it happened.

With the proxy and a little script we can track what happened in all open transactions before the deadlock happened. A classic example is the cross-over deadlock:

[36] received a ERR(1213, Deadlock found when trying to get lock; try restarting transaction), dumping all active transactions
  [35].1: select * from commit where id = 1 for update
  [35].2: select * from commit where id = 2 for update
  [36].1: select * from commit where id = 2 for update
  [36].2: select * from commit where id = 1 for update

35 and 36 are the two transactions we have open, the last statement in 36 triggered the deadlock.

In the script we stored …

No, these slides are not fresh from the User Conference in Santa Clara… ?

Today, I held a presentation in front of all developers and support engineers of our technical department about database partitioning, MySQL Proxy, HSCALE and the progress we are making.

Download the presentation slides here.

As written here and here I’ve been working on a MySQL Proxy Lua module that transparently splits up tables into multiple partitions and rewriting all queries to go to the right partition.

I finally got everything together to release a 0.1 version. Go on and download, try and read more about HSCALE 0.1.

All this started out as a prototype just to see if it could be done. And after adopting parts of our main product to use partitions via HSCALE + MySQL Proxy (which was an easy task, we just had to rewrite a few out of hundreds of statements) I really think that this could work out in a larger scale.

What Will Come Next?

Just a …

Yesterday I went through hell while testing the MySQL Proxy partition Lua scripts I am working on in a high concurrency environment. I am sending multiple queries to the server and build up a combined result set in read_query_result - something like this. The proxy returned weird results complaining about multiple result sets being sent from time to time at totally different places even though I was sure that for each query only one result set has been sent. And some of the results were just wrong, a lot of our tests failed unexpectedly.

After long hours flicking almost every switch I simply removed the Lua module declaration from the main LUA script passed to the proxy via proxy-lua-script since it was the only thing left that …

One of the workshops on our Barcamp two weeks ago had to do with the MySQL-Proxy from Jan Kneschke.

Yet, we found out, that the proxy is rather unuseable for our task. Read here why.

Continue reading "Playing arround with the MySQL-Proxy on Mayflower-barcamp"