After a discussion with my colleague Stephane Varoqui we decided to see how Gearman and the NDBAPI could be used together. The result of the POC was a Gearman worker and a couple of clients (clients and workers use Google Protocol Buffers as the protocol). The worker can:

• set/get/delete records on a single table in MySQL Cluster using the primary key
• set/get/delete "any" type. It is not possible to dynamically add types but this is done at compile time.
  
• supports the following SQL data types: (UNSIGNED) INTEGER, (UNSIGNED) BIGINT, CHAR, VARCHAR/VARBINARY
• supports the following Google Protocol Buffer …

Cassandra is fast emerging as one of the key NoSQL databases.  While we often express that the point of NoSQL is to offer more choice than an “RDBMS” hammer for every nail, there are practical reasons why a small number of stack technologies gain dominance and others circle on the sidelines.

Cassandra has already ticked many of the boxes needed to shoot it into the stratosphere as a widely used, default database platform.  Especially so in the web world where high scalability, high availability, open source and being proven by a bigger fish all matter.  Specifically Cassandra has:

• The ability to scale across many nodes
• The ability to scale to many hundreds of gigabytes of data
• High availability, losing a node doesn’t take down the cluster & online node provisioning and data distribution (and …

As long as the whole world is chasing this meaningless “NoSQL” buzzword, we should recognize that InnoDB is usable as an embedded database without an SQL interface. Hence, it is as much of a NoSQL database as anything else labeled with that term. And I might add, it is fast, reliable, and extremely well-tested in the real world. How many NoSQL databases have protection against partial page writes, for example?

It so happens that you can slap an SQL front-end on it, if you want: MySQL.

Related posts:

1. On the unhelpfulness of NoSQL My favorit

2. A Review of Beginning Database Design by Clare Churcher

   Beginn

…

Peter took a look at Redis some time ago; and now, with the impending 1.2 release and a slew of new features, I thought it time to look again.

One of the more interesting features in 1.2 is the ability to operate in "append-only file persistence mode", meaning Redis has graduated from a semi-persistent to a fully-persistent system! Using the redis-benchmark script included, I ran the following command

./redis-benchmark

in five modes:

1 - In-Memory
I set "save 900000000 900000000" so nothing would be written to disk during the tests.
2 - Semi-Persistent
I set "save 1 1" so that changes would be flushed to disk every second (assuming there was at least one change the previous second).
…

The No-SQL tag really lumps together a lot of concepts that are in fact as distinct from eachother as they are from SQL/RDBMS.

An object store is not at all similar to Cassandra and Hypertable, which is not at all like an column store. And when looking at BigTable derivatives, it’s quite important to realise that Google actually does joins in middle layers or apps, so while BigTable does not have joins, the apps essentially do use them – I’ve heard it professed that denormalising everything might be a fab idea, but I don’t quite believe in that for all cases, just like I don’t believe in ditching the structured form of RDBMS being the solution.

SQL/RDBMS has had a few decades of dominance now, and has thus become the great “general purpose” tool. With the ascent of all the other tools, it’s definitely worthwhile to look at them, but also realise that each (inluding SQL based ones) have their place. Moving all your …

OpenSQLCamp was a huge success! I took videos of most of the sessions (we only had 3 video cameras, and 4 rooms, and 2 sessions were not recorded). Unfortunately, I was busy doing administrative stuff for opensqlcamp for the opening keynote and first 15 minutes of the session organizing, and when I got to the planning board, it was already full….so I was not able to give a session.

• 
  
• Comparing Non-Relational Databases: MongoDB, Tokyo Tyrant, CouchDB by Igal Koshevoy of Pragmaticraft

OpenSQLCamp was a huge success! Not many folks have blogged about what they learned there….if you missed it, all is not lost. We did take videos of most of the sessions (we only had 3 video cameras, and 4 rooms, and 2 sessions were not recorded).

All the videos have been processed, and I am working on uploading them to YouTube and filling in details for the video descriptions. Not all the videos are up right now….right now all the lightning talks are up.

• 
  
• All the lightning talks belowin one continuous video (just over 54 minutes total)
  
• The Graph Engine (Antony Curtis)
  
• …

Last week I spent some time speaking with Kevin Weil, head of analytics at Twitter. Twitter, from a technology perspective, has had a bit of a hard time due to their stability issues in their early days.  Kevin was keen to point out that he feels this was due to the incomparable growth Twitter was experiencing at the time and their constant struggle to keep up.  Kevin was also keen to show that Twitter prides themselves on striving for engineering excellence, the creation & contribution to new technologies and generally assisting in pushing the boundaries forward.  Our conversation naturally centered on analytics at Twitter.

Twitter, like many web 2.0 apps, started life as a MySQL based RBDMS application.  Today, Twitter is still using MySQL for much of their online operational functionality (although this is likely to change in the near future – think …

No I did NOT post the following comment

This is ridiculous. You didn't test MySQL, you tested a failing DNS lookup on authentication.

resulting in the follow up post :

My previous post Redis, Memcache, Tokyp Tyrant, MySQL comparison had a flaw as pointed out by this comment. The MySQL was taking a huge time for doing a reverse DNS lookup.

But as always ... Everything is just a Freaking dns problem :)

Technorati Tags: dns dnsproblem mysql …

This is part 3 of my Tyrant extra's, part 1 focused on durability, part 2 focused on the perceived performance wall.

#3.  Tokyo Cabinet Can have only a single writer thread, bottlenecking performance

When writing an application using Tokyo Cabinet only one connection can be opened as a “writer”  while the rest are readers.  Tyrant allows for multiple “writes”  to be sent in from multiple applications but it still single threads them when writing out to disk.   If you run several threads all just inserting into Tyrant your will see tyrant hit 100% Cpu on 1 core, and your writes will start to peter out quickly.

In my tests when I was not disk bound (FS Cache writes) I was able to complete 4Million inserts in a little over 91 seconds using 8 threads.  I actually averaged 43896.98 inserts per second during my 8 thread test.  Moving to 10 threads doing the same 4Million inserts I …