This past week I have been working with 2 Web2.0 startups - members of Sun Startup Essentials, they get the support for free - that are looking at improving the response time of their web site.

We've been working at understanding how they servers behave today and what could prevent their application to scale, which led me to show how to run a quick diagnostic on an existing system and to restate some key fundamentals in terms of scalability.

Let's start by the fundamentals.

Any piece of software is nothing else than a set of instructions that need 2 things: computing units to execute on, and a fast access to data.

Computing units are available in the form of a hardware threads located on a processor. The number of threads that a server makes available to the application depends on the number of processors on the server, the number of …

With many thanks to Vadim at Percona for his analysis of different capabilities of different columnar dbms. Definitely good information, and well documented.  Queries and results available at:

http://www.mysqlperformanceblog.com/2009/11/02/air-traffic-queries-in-infinidb-early-alpha/

Of course InfiniDB, does offer multi-threaded processing for all offerings and distributed prRead More...

With many thanks to Vadim at Percona for his analysis of different capabilities of different columnar dbms. Definitely good information, and well documented.  Queries and results available at:

http://www.mysqlperformanceblog.com/2009/11/02/air-traffic-queries-in-infinidb-early-alpha/

Of course InfiniDB, does offer multi-threaded processing for all offerings and distributed processing (enabled with Enterprise Edition additional functionality) that was beyond the scope of hisRead More...

Today I’m proud to announce the first public release of our ActiveRecord database connection magic plugin: DbCharmer.

DB Charmer – ActiveRecord Connection Magic Plugin

DbCharmer is a simple yet powerful plugin for ActiveRecord that does a few things:

1. Allows you to easily manage AR models’ connections (switch_connection_to method)
2. Allows you to switch AR models’ default connections to a separate servers/databases
3. Allows you to easily choose where your query should go (on_* methods family)
4. Allows you to automatically send read queries to your slaves while masters would handle all the updates.
5. Adds multiple databases migrations to ActiveRecord

Installation

There are two options when …

You want to use or to know more about Sun technologies? Join these Webinars to improve your knowledge and skills. You will be able to ask questions to Sun's experts.

• Mercredi 10 Février : Securité pour les applications Web. Pour les startups du Web, la protection et la sécurisation de leurs applications, de leurs données, et de celles de leurs clients est un véritable facteur clé de succès.  Ce Webinar couvre les différents challenges liés à la sécurité ainsi que les solutions associées telles que l'encryption, l'authentification, les certificats, la sécurisation du stockage et le stockage à tolérance de panne, les environnements étanches. Les architectes de Sun Startup Essentials présenteront des implémentations économiques basées sur des composents standards et ouverts tel qu'Apache, MySQL et ZFS. Ce webinar est …

You want to use or to know more about Sun technologies? Join these Webinars to improve your knowledge and skills. You will be able to ask questions to Sun's experts.

• Mercredi 10 Février : Securité pour les applications Web. Pour les startups du Web, la protection et la sécurisation de leurs applications, de leurs données, et de celles de leurs clients est un véritable facteur clé de succès.  Ce Webinar couvre les différents challenges liés à la sécurité ainsi que les solutions associées telles que l'encryption, l'authentification, les certificats, la sécurisation du stockage et le stockage à tolérance de panne, les environnements étanches. Les architectes de Sun Startup Essentials présenteront des implémentations économiques basées sur des composents standards et ouverts tel qu'Apache, MySQL et ZFS. Ce webinar est …

A couple of days ago I posted about scaling writes in mysql. I didn't say much about read performance in that post because a) it was irrelevant at the time, and b) there are thousands of articles all over the web that already cover read performance.

In this post I'm going to cover some of the things that I did to improve read performance for my own application. It may be relevant to others, however, you'd still need to read a whole bunch of other articles to understand MySQL read performance.

Looking at our access patterns, it turned out that there were two classes of read queries.

1. Reads to build the daily summaries
2. Reads from the summary tables in response to user actions

The former dealt with far more data at one go, but was only run once. Queries for this pattern were slow …

We use MySQL on most of our projects. One of these projects has a an access pattern unlike any other I've worked on. Several million records a day need to be written to a table. These records are then read out once at the end of the day, summarised and then very rarely touched again. Each record is about 104 bytes long (thre's one VARCHAR column, everything else is fixed), and that's after squeezing out every byte possible. The average number of records that we write in a day is 40 million, but this could go up.

A little bit about the set up. We have fairly powerful boxes with large disks using RAID1/0 and 16GB RAM, however at the time they only had 4GB. For BCP, we have a multi-master set up in two colos with statement level replication. We used MySQL 5.1.

My initial tests with various parameters that affect writes showed that while MyISAM performed slightly better than InnoDB while the tables were small, it quickly …

We ran a quick scalability test of Calpont join behavior across using a Star Schema Benchmark data set at a scale factor of 1000. The Star Schema Benchmark transforms a TPC-H / DBT-3 data to a more standardized data warehouse star schema data model, and the 1000 scale factor includes 6 billion rows in the primary fact table. Information on the star schema bench (SSB) can be found at http://www.cs.umb.edu/~xuedchen/research/publications/DataWarehousePerformanceDissertationProposal.pdf .

-----------------------------------------------------------------------------------------------------------------------------------------
-- Note that these queries are run without any tuning or indices created for these joins or filters.
-- Basically, this is just 1) Create tables (without index or …

My old friend and collaborator Theo Schlossnagle at OmniTI posted his slides from his Scalable Internet Architectures talk at VelocityConf 2009.

The slides are brilliant even without seeing Theo talk and I highly recommend the time it takes to flip through them, for anyone who is interested in systems performance. If anyone took an mp3 of this talk I’m dying to hear it, please let me know.

For those of you unfamiliar with OmniTI, Theo is the CEO of this rather remarkable company specializing in Internet-scale architecture consulting. They generalize on Internet-scale architecture, not on one specific dimension the way Pythian specializes on the database tier. This allows them to see Internet-scale workloads from a unique systemic, multidisciplinary point of view; from the user experience all the way up the stack, through …