Often I think about challenges that organizations face with “Big Data”.  While Big Data is a generic and over used term, what I am really referring to is an organizations ability to disseminate, understand and ultimately benefit from increasing volumes of data.  It is almost without question that in the future customers will be won/lost, competitive advantage will be gained/forfeited and businesses will succeed/fail based on their ability to leverage their data assets.

It may be surprising what I think are the near term challenges.  Largely I don’t think these are purely technical.  There are enough wheels in motion now to almost guarantee that data accessibility will continue to improve at pace in-line with the increase in data volume.  Sure, there will continue to be lots of interesting innovation with technology, but when organizations like …

Intent Media

Issue addressed: Ad hoc analytics on clickstream data arriving too fast for InnoDB or NoSQL to handle.

TokuDB powers an online advertising application

The Company: Headquartered in New York, Intent Media is a fast-growing online advertising startup. The company helps some of the largest online retailers monetize their traffic more efficiently at scale by showing highly relevant and targeted advertising to the 97+% of e-commerce visitors who do not transact.

The Challenge: The Intent Media platform processes hundreds of millions of events a day generated by media placements across leading e-commerce sites — a textbook “Big Data” challenge. Intent Media’s data is used to optimize media placements, drive segmentation models, and …

Reading yesterday that the NSA has submitted a proposal to Apache to incubate their Accumulo platform.  This, according to the description, is a key/value store built over Hadoop which appears to provide similar function to HBase except it provides “cell level access labels” to allow fine grained access control.  This is something you would expect as a requirement for many applications built at government agencies like the NSA.  But this also is very important for organizations in health care and law enforcement etc where strict control is required to large volumes of privacy sensitive data.

An interesting part of this is how it highlights the acceptance of Hadoop.  Hadoop is no longer just a new technology scratching at the …

We don’t pay enough attention to Hadoop.

By “we” I mean DBAs, the rest of the world is paying plenty of attention to Hadoop. Recently, I started asking my customers and fellow DBAs about Hadoop adoption in their company. Turns out that many of them have Hadoop. Hadoop shows up in large companies and small ones, in established industries and in startups. Its everywhere.

The way Hadoop shows up in all companies, and the way DBAs don’t pay Hadoop much attention, reminds me a lot of how MySQL started showing up in the enterprise. It didn’t start by DBAs showing up one morning and telling their managers:
“There’s this new open source database. Its not as stable as Oracle and it doesn’t have all the features we need, but man – its going to save us tons of money, and its pretty simple to manage.”

Nope, this never happened. What happened instead is that developers learned about MySQL, and it seemed …

MySQL Cluster* is one of the most advanced and scalable databases available today and, despite what its name might suggest, it is also a brilliant NoSQL database**.
Let me discuss this statement!
First, let’s discuss the high level issues that NoSQL databases try to address: -     Scalability. Traditional RDBMS technology was designed four decades ago, and is not appropriate for today’s Big Data requirements. Database systems today need to be able to scale horizontally over multiple machines to handle millions of users. As the CAP theorem states, it is not possible to achieve availability, scalability and consistency in one system. Several NoSQL databases sacrifice consistency for availability and scalability. -     RDBMS has a rigid data model. Once a normalized data model has been defined in an RDBMS, it is …

Just wanted to record for the history books that:


drizzle> select js_eval('var d = new Date(); "Drizzle started running JavaScript at: " + d;')\g
+----------------------------------------------------------------------------------+
| js_eval('var d = new Date(); "Drizzle started running JavaScript at: " + d;') |
+----------------------------------------------------------------------------------+
| Drizzle started running JavaScript at: Mon Aug 29 2011 00:23:31 GMT+0300 (EEST) |
+----------------------------------------------------------------------------------+
1 row in set (0.001792 sec)


I will push this onto launchpad tomorrow, after a good nights sleep and final code cleanups.

read more

A friend recently bought a GM car. I proceeded to inform him that I am shorting GM stock (technically a put option). He was shocked. “But they make great cars,” he exclaimed. I responded, “I’m not shorting the cars, I’m shorting the company.” Why am I recounting this exchange? Because I believe that the new wave of NoSQL companies—as opposed to the rebranded ODBMS—presents the same situation. I am long the products, but short the companies.
Let me explain. NoSQL companies have built some very cool products that solve real business problems. The challenge is that they are all open source products serving niche markets. They have customer funnels that are simply too small to sustain the companies given their low conversion/monetization rates.
These companies could certainly be tasty acquisition targets for companies that actually make money. But as standalone companies, sadly, I would short them. On that note, I am off to …

Engine Yard acquires Orchestra. Red Hat considers NoSQL move. And more.

# Engine Yard announced a definitive agreement to acquire Orchestra, bringing PHP expertise to the Engine Yard platform.

# Red Hat’s CEO indicated the company is interested in a NoSQL or Hadoop acquisition.

# Gluster announced Apache Hadoop compatibility in the next GlusterFS release.

# Microsoft signed an agreement with China Standard Software Co (CS2C) to …

InternetNews.com yesterday published an article based on an interview with Red Hat CEO Jim Whitehurst asking the question “Is Red Hat Interested in the Database Market?”

In truth there was no real need to ask the question, as Whitehurst’s comments made it pretty clear that Red Hat is interested in the database market, and specifically the NoSQL database market.

“When I say I don’t want to be a database company, I’m saying that I don’t want to be a SQL database company,” Whitehurst said.

In case the implications of that statement were not entirely clear, he later added:

“But we would be very interested in a NoSQL type database or Hadoop type thing,” Whitehurst said. “Those are interesting as they represent net new.”

The article adds that Whitehurst would not …

I just uploaded my slides from my Open DB Camp / Froscon talk to Slideshare:

read more