Departmental or subject-specific data warehouses – known as “data marts” in the industry – seem to be gaining in popularity.  Fueled partly by companies wanting to start small with focused projects in today’s economy, and partly by advances in data warehousing technology improving affordability and deployability, data marts seem to be popping-up everywhere.

In most cases, data mart projects are driven by the head of a business unit or a functional group (like Sales) needing to analyze their own slice of data in order to run their department more efficiently and effectively.  The data may come directly from an operational system or a combination of source systems resulting in what’s called an “independent data mart”, or it may come directly from a larger, enterprise data warehouse in a hub-and-spoke or “dependent data mart” configuration.

In either case today, according to industry analysts, companies …

Departmental or subject-specific data warehouses - known as "data marts" in the industry - seem to be gaining in popularity.  Fueled partly by companies wanting to start small with focused projects in today's economy, and partly by advances in data warehousing technology improving affordability and deployability, data marts seem to be popping-up everywhere.

In most cases, data mart projects are driven by the head of a business unit or a functional group (like Sales) needing to analyze their own slice of data in order to run their department more efficiently and effectively.  The data may come directly from an operational system or a combination of source systems resulting in what's called an "independent data mart", or it may come directly from a larger, enterprise data warehouse in a hub-and-spoke or "dependent data mart" configuration.

In either case today, according to industry analysts, companies are looking for data …

Why was Teradata able to become the leader of data warehousing at the super high-end (e.g. greater than 25 TB’s)?  Why was Netezza only the second pure-play data warehousing company to go public by focusing on the 10 – 25 TB range of opportunities?  Why did Oracle after so many years of denial finally announce a joint hardware / software product for data warehousing with HP, the Exadata data warehouse server?  Why did Microsoft acquire DATAllegro, one of the earlier data warehousing appliances? Why are there now dozens of data warehouse appliances available on the market today, and – more importantly – how should a customer choose which one to purchase? 

In all these cases, the vendors have listened to the market and concluded that the most optimal way to serve the customer is through a true data warehouse appliance.  Given that there are so many flavors of appliances, though, here are some things to …

The Kickfire MySQL Appliance is offically launched!

We just announced today, along with a new customer, and strategic partnerships with ten leading service companies including Percona, the MySQL performance experts.

Look for more news next week from Kickfire as we head into the MySQL conference. Kickfire will also give a keynote on the first day of the conference and will make a surprise announcement! Stay tuned …

At the March Boston MySQL User Group meeting, Jacob Nikom of MIT’s Lincoln Laboratory presented “Optimizing Concurrent Storage and Retrieval Operations for Real-Time Surveillance Applications.” In the middle of the talk, Jacob said he sometimes calls what he did in this application as “real-time data warehousing”, which was so accurate I decided to give that title to this blog post.

The slides can be downloaded in PDF format (1.3 Mb) at http://www.technocation.org/files/doc/Concurrent_database_performance_02.pdf. The 54 minute video can be downloaded (644Mb) at http://technocation.org/node/693/download or streamed directly in your browser at http://technocation.org/node/693/play. …

We just shipped and installed the Kickfire appliance in the data center of our first web 2.0 customer this week. We’re very excited about this new customer. With already over a million active members, this company continues to grow in spite of a challenging economic environment because it has a clearly defined audience and a business model which adds value to its members while adding money to its coffers. Part of the value add to their member base comes from well-targeted discount and coupon offers. In order to achieve this, the company runs complex analytics to understand members’ behaviors and responses and uses this data to help its advertising customers better target their offers.

As with many web 2.0 companies, this customer has built its application on MySQL. MySQL has helped them scale their web application well but was presenting performance and scalability challenges for their analytics. With their fact table in the …

After receiving an email about talking with Robert David, Director of Sales at KickFire, I checked out recent news on KickFire, as there had been little written about this company aside from the big splash they made at the MySQL Users Conference back in April 2008.And, lo and behold, there was a piece of news, posted on October 14 - Kickfire Enters Into MySQL Enterprise Agreement With Sun

If there is one thing that a DBA or data warehouse architect can count on, it is that data volumes will increase while budgets will decrease.

This is why MySQL 5.1 and its partitioning capabilities are so interesting. I’m going to demonstrate how you can build a small/medium-sized data warehouse or data mart (1-10 TB range) on a shoe-string budget.

the mission

I decided to convert a relatively large statistics table (750m rows, 140GB in size in about 10 partitions) on a test machine from MyISAM to the Archive storage engine. After a long conversion process, my data, on disk, ended up being about 21GB, for an impressive compression ratio of 6.7:1.

Prior to MySQL 5.1, one of the drawbacks to the archive storage engine was that you could not index it; however, with partition pruning, you can get yourself a “free” index on a large archive table by splitting it into date-based chunks, whether by …

When your company decides that "it is time to build a data warehouse", what thoughts come to mind?1) A magical fairy ice cream land where data is presented in chocolate shells for everyone to digest perfectly;2) A big literal warehouse in the industrial section of town with rusty old containers;3) Another place to put data, which means another place for you to track and monitor additional