Note – this post is a footnote for my MySQL users conference presentation Sharding Using Spockproxy: A Sharding-only Version of MySQL Proxy.

When you finished my last blog post you have a directory for each shard full of load files. The tables had been created so now we’re ready to load these files.

These two queries will generate load scripts, modify them as needed or write your own.

SELECT concat(‘ LOAD DATA INFILE \’
/db0′, ‘/’, st.table_name, ‘\’ INTO TABLE ‘, st.table_name, ‘;’) AS ”
FROM shard_table_directory st
WHERE status = ‘universal’;

and run this once for each shard (change the sr.database_id = 1 to each database_id).

SELECT concat(‘ LOAD DATA INFILE \’
/db’, database_id, ‘/’, st.table_name, sr.range_id, ‘\’ INTO TABLE ‘, …

Note – this post is a footnote for my MySQL users conference presentation Sharding Using Spockproxy: A Sharding-only Version of MySQL Proxy.

During the development and testing of our Spockproxy I found I was dumping and loading the data from our old non-sharded databases into shards repeatedly while we tried to get the configuration correct. I developed this process and I’ve found that id works easily; hopefully you’ll find it helpful.

1. set up a directory that your database server can dump to – in it create one directory for each shard and name them ‘db1’, ‘db2’, ‘db3’ and so on and a directory for the universal db as ‘db0’.

2. modify this SQL, change the
to the path to the directory you created in step 1 (the parent directory) and change the 30 where it says “AND range_id < 30” to be the …

Dump and load the schemas (one for universal and one for the shards).

Note – this post is a footnote for my MySQL users conference presentation Sharding Using Spockproxy: A Sharding-only Version of MySQL Proxy.  

Dumping the schema is fairly straight forward using mysqldump.  There are two issues: we need to separate the “universal” tables from the sharded tables and you need to decide what to do about auto_increment.

Separating the tables is simple if you have your shard_table_directory table set up the way you like it then run:

SELECT group_concat(table_name, SEPARATOR ‘ ‘) FROM shard_table_directory WHERE status = ‘federated’;

and

SELECT group_concat(table_name, SEPARATOR ‘ ‘) FROM shard_table_directory WHERE status = ‘universal’;

and keep …

Do you want a SPOCK tee shirt?  Read on:

I’m going to give a talk on Spockproxy (a sharding / connection pooling only version of MySQL proxy) at the MySQL conference and as I prepare I’m looking to give my talk broad appeal and try to address all kinds of problems folks might have sharding their databases.

So I’m throwing this question out to the MySQL community – Have you looked into sharding your database(s)?  Did you come up against problems that were difficult to solve? Please take a moment and let me know about them.  I’d like to address how to fix them with Spockproxy.  Even if you’ve solved these issues already or have no intension of using Spockproxy your problems could be interesting to others; add your sharding problem(s) in the comment below and look for me  …

Last week I got a question about sharding using our Spockproxy.  The question was how can I create a query for the proxy so it effectively runs:

/*in shard 1*/
SELECT * FROM table_a WHERE f_key IN (a, b, c);

UNION

/*in shard 2*/
SELECT * FROM table_a WHERE f_key IN (d, e, f);

By design our proxy will not do this.  The whole point is to hide the sharding from the application.  Given a query it will either send the same query to all the shards and combine the results or only send that query to one shard when it can figure out that the results(s) can only come from one shard (because you specified the shard key in the where clause).

I did figure out a way it could be done using views but would this ever be desirable?  

Like “Hard Coding” where values are built into the code of your application I’ll call this technique “Hard …

I’ve reposted the slides from my Spockproxy talk (Spockproxy is a Sharding only version of the MySQL proxy).  Since I’ve have to move this web site it’s been some effort to move all of the files into their new homes.

These slides are in a variety of formats and have loads of great information if you’re considering a sharding solution – even if that is not Spockproxy.  Of course once you see how easy it is you’ll put it on your short list.

The slides are available at http://www.frankf.us/projects/spockproxy/  If you want to hear the talk you’ll have to invite me.