How do you get data from a MySQL Cluster into Hadoop? Easy, replicate from the cluster to a stand alone MySQL instance and from there use the MySQL Hadoop Applier to HDFS.
This question came from a long time MySQL user who has jumped into the Big Data world.
MaxScale is a Proxy for the MySQL protocol built with a modular architecture. The underlying concept of modules allows to extend the MaxScale proxy services. The current version implements Read Write splitting and Connection Load Balancing. Internally MySQL queries go through a SQL parsing phase. This gives MaxScale great capabilities regarding queries routing.
So if [...]
I blogged a couple of weeks ago about my upcoming MySQL/Hadoop talk at SCaLE 12X.
Unfortunately I had to cancel. A few days after writing the
article I came down with an eye problem that is fixed but
prevents me from flying anywhere for a few weeks. That's a pity
as I was definitely looking forward to attending the conference
and explaining how Tungsten replicates transactions from MySQL
into HDFS.
Meanwhile, we are still moving at full steam with Hadoop-related
work at Continuent, which is the basis for the next major
replication release, Tungsten Replicator 3.0.0. Binary builds and
documentation will go up in a few days. There will also be many
more public talks about Hadoop support, starting in April at
…
We are in the middle of switching to GTID based replication from the good old logfile & log position based replication.
But what is GTID? GTID is an abbreviation of ‘GLOBAL TRANSACTION ID’ what speaks for itself: each transaction of a mysql cluster got its globally unique transaction ID, and the DBA have not spend time with positioning slaves, as well as we don’t have to ‘freeze’ any of the servers because of a master failover. The only thing we have to care about: to know what server should be used as a replication master.
OK, what was the problem with the old file-based replication?
Check out this scenario: you have two master servers, called db-master1 and db-master2 and two slaves db-slave1 db-slave2. All the writes are happening on db-master1 and this will replicating to the slaves and the failover master too.
So you have your replication set up, and everything works well. But there is a point …
[Read more]On the road again - FOSSAsia
It has been a few busy months until now. I have moved from Italy to Thailand, and the move has been my first priority, keeping me from attending FOSDEM and interacting with social media. Now I start catching my breath, and looking around for new events to attend. But before I get into this, let’s make a few things clear:
I have just implemented MASTER_GTID_WAIT() in MariaDB 10.0. This can be used to give a very elegant solution to the problem of stale reads in replication read-scaleout, without incuring the overheads normally associated with synchronous replication techniques. This idea came up recently in a discussion with Stephane Varoqui, and is similar to the concept of Lamport logical clock described in this Wikipedia article.
I wanted to describe this, hoping to induce people to test and maybe start using this, as it is a simple but very neat idea, actually.
A very typical use of MariaDB/MySQL …
[Read more]If you happen to work with personal data, chances are you are subject to SOX (Sarbanes-Oxley) whether you like it or not.
One of the worst aspects of this is that if you want to be able to analyse your data and you replicate out to another host, you have to find a way of anonymizing the information. There are of course lots of ways of doing this, but if you are replicating the data, why not anonymize it during the replication?
Of the many cool features in Tungsten Replicator, one of my favorites is filtering. This allows you to process the stream of changes that are coming from the data extracted from the master and perform operations on it. We use it a lot in the replicator for ignoring tables, schemas and columns, and for ensuring that we have the correct information within the THL.
Given this, let’s use it to anonymize the data as it is being replicated so that we don’t need to post-process it for analysis, and …
[Read more]MySQL 5.6 has been GA for just over a year now. See MySQL 5.6.10 Release Notes. Congratulations on your birthday! That is quite a long time. I was using it earlier in production because it worked and could do things that 5.5 could not do, but earlier versions were to use at your own risk, and indeed if prodded incorrectly would fall on the floor. That is fair enough because they were work in progress, yet if you poked them the right way they did a very good job. Those dev versions have been long since upgraded which is good so they do not need quite as much care and attention.
So from where I see 5.6 it works very well. One big change that has made a large difference but which I think a lot of people may not really understand or use is the …
[Read more]
It's my pleasure to be presenting at SCaLE
12X on the subject of real-time data loading from MySQL to Hadoop.
This is the first public talk on work at Continuent that
enables Tungsten Replicator to move transactions from
MySQL to HDFS (Hadoop Distributed File System). I will
explain how replication to Hadoop works, how to set it up, and
offer a few words on constructing views of MySQL data using tools
like Hive.
As usual with replication everything we are doing on Hadoop
replication is open source. Builds and documentation will
be publicly available …
At FOSDEM 2014 ProxySQL and MaxScale were both presented. Both are proxy that can help build sophisticated MariaDB/MySQL architectures. But currently what is the most used proxy with MySQL? It is HAproxy. HAproxy is a level 4 proxy that has no knowledge of the MySQL protocol. Being low level makes it very fast but it [...]