At the MySQL Conference & Expo 2008, Britt Crawford and Justin McCarthy, both from Cafepress.com, gave us a very interesting talk on scaling with HiveDB. I took a few notes (pasted below), their slides are online (warning: 6.1MB PDF), and if you’re after their abstract its available as well.

I also took a video of them (refer to Slide 12, for the IRC conversation):

The quick notes:

• OLTP optimised (as it serves cafepress.com)
• Cannot lock tables, or take it offline
• Constant response time is more important than low latency (little slower query is ok, just not exponentially …

All of the videos from the 2008 MySQL Conference have been processed and uploaded. Links to the videos, slides, notes, photos for each presentation are all on the mega-conference page at:
http://forge.mysql.com/wiki/MySQLConf2008Notes

This represents many hours of my own toil, but it also reflects plenty of people who have blogged, edited the wiki pages and speakers who wrote and gave tutorials and presentations. I am proud of everyone’s efforts to offer so many learning resources for free….

Enjoy! EDIT: I forgot to thank Jay, the folks at O’Reilly and all the speakers for giving me explicit permission to video and freely offer their presentations.

If you know of any video, audio, notes, slides, photos, etc that are not linked, please link them at the wiki page. If you can’t or won’t, please comment here and I will update the wiki …

I did an interview with Barton George from Sun while I was at the conference last week. Barton has now posted the interview. If you’re quick, you can listen to it before I do.

Topics: everything and anything, including Maatkit and PostgreSQL.

Baron Schwartz, Barton George, maatkit, mysqluc08, mysqluc2008, Podcast, …

Here’s a rundown of Thursday (day 3) of the MySQL Conference and Expo. This day’s sessions were much more interesting to me than Wednesday’s, and in fact I wanted to go to several of them in a single time slot a couple of times.

Inside the PBXT Storage Engine

This session was, as it sounds, a look at the internals of PBXT, a transactional storage engine for MySQL that has some interesting design techniques. I had been looking forward to this session for a while, and Paul McCullagh’s nice explanations with clear diagrams were a welcome aid to understanding how PBXT works. Unlike some of the other storage engines, PBXT is being developed in full daylight, with an emphasis on community involvement and input. (Indeed, I may be contributing to it myself, in order to make its monitoring and tuning capabilities second to none).

PBXT has not only a unique design, but a …

Here is the quick notes from the session scaling heavy concurrent writes in real time by Dathan Pattishall. Its bad that he left Flickr i.e. Yahoo. Hopefully they will find a replacement, if not contact me I have few people who are interested.

• Who am I
  • since 1999 working on mysql
  • scaled many companies (FriendFinder, Friendster, Flickr, now RockYou)
  • Favorites federation, partitioning, shards, RAID-10
• Requirements
  • scale better
  • store data forever
  • associate time with the data
  • allow for …

Here is the quick notes from the session Helping InnoDB scale on servers with many cores by Mark Callaghan from Google (mcallaghan at google dot com).

• we have a team now, to help scale MySQL to do the enhancements (9 people,  I hope yahoo management reads this)
• Overview
  • describe the problems on big servers
  • work done by InnoDB community
  • ask MySQL/InnoDB to fix the problems by taking the patches
• Community team
  • InnoDB/Oracle
  • Google MySQL team
  • InnoDB community
  • Percona - Peter and Vadim
…

The Future of MySQL by Monty Widenius and Jay Pipes.

• Why this talk
  • MySQL and Sun should become more transparent
  • Easier to discuss and act when you have facts
  • when user know the limitations, they know how to go around
• Threads
  • one connection/thread doesn’t work well
  • no priority threads
  • no way to ensure we have X threads
• Symptoms
  • Too many context switches
  • we are not using multi-core efficiently
  • does not scale that well after 4-8 cores
• Solution
  • –thread-handling=pool-of-threads (6.0) …

Day two of the conference was a little disappointing, as far as sessions went. There were several time blocks where I simply wasn’t interested in any of the sessions. Instead, I went to the expo hall and tried to pry straight answers out of sly salespeople. Here’s what I attended.

Paying It Forward: Harnessing the MySQL Contributory Resources

This was a talk focused on how MySQL has made it possible for community members to contribute to MySQL. There was quite a bit of talk about IRC channels, mailing lists, and the like. However, the talk gave short shrift to how MySQL plans to become truly open source (in terms of its development model, not its license). I think there was basically nothing to talk about there. I had a good conversation about some of my concerns with the speaker and some others from MySQL right afterwards.

There was basically nobody there — I didn’t count, but I’d say maybe 10 or 12 people. I …

From Jeff Rothschild, VP of technology, Facebook

1. The power of connectedness
2. Impact on the database
3. Our challenge 

• The power of connectedness
  • photo tagging
  • getting an email that someone tagged a photo in Facebook
  • inter tagging between friends and it continues as a network
  • outstanding growth because of the photo tagging
  • 26B photos in archive now
  • Most trafficked photos applications than anybody else
  • events, invites impact on social graph
  • Opening up the social graph to outside and create a platform and API, so other developers can make use of it (28K applications so far)
  …

Here is the list of MySQL benchmarking tools that were discussed during the "Benchmarking tools" session today.

• mysqlslap (5.1 onwards)
• sql-bench
• supersmack
• ApacheBench
• MySQL benchmark() function
• mybench
• …