Although the main interface between applications and a Vitess database is through the MySQL protocol, Vitess is a large and complex distributed system, and all the communication between the different services in a Vitess cluster is performed through GRPC. Because of this, all service boundaries and messages between Vitess' systems are specified using Protocol Buffers. The history of Vitess' integration with Protocol Buffers is rather involved: We have been using and keeping up to date with the Go Protocol Buffers package since its earliest releases, up until May last year, when Google released a new Go API for Protocol Buffers, which is not backwards compatible with the previous Go package.

Cross posting link Golang is a wonderful language. It's simple, and most of the time not confusing or surprising. This makes it easy to jump into library code and start reading and quickly understand what's going on. On the other hand, coming from other languages, there are a few features that would make our lives easier. We are building Vitess using mostly golang, and most of us are happy with this choice.

Writing MySQL Proxy in GO for self-learning: Part 2 — decoding handshake packet

After we built a generic TCP proxy, we can continue with our journey. Today’s goal will be to understand the MySQL Protocol, receive, decode, encode and send to the client the first packet sent by the MySQL Server.

MySQL connections threads

Each client connection to MySQL Server handled by a thread. MySQL is portable, so the underhood threads implementation is system dependent (Windows, macOS, and Linux have their own threads implementation).

What important to us, it’s to understand that a single client connection …

Version 0.3.0 of dbdeployer has gained the ability of deploying multiple sandboxes concurrently. Whenever we deploy a group of sandboxes (replication, multiple) we can use the --concurrent flag, telling dbdeployer that it should run operations concurrently.

What happens when a single sandbox gets deployed? There are six sets of operations:

1. Create the sandbox directory and write down its scripts;
2. Run the initialisation script;
3. Start the database server;
4. Run the pre-grants SQL commands (if any;)
5. Load the grants;
…

How it happened

A few years ago I started thinking about refactoring MySQL-Sandbox. I got lots of ideas and a name for the project (dbdeployer) but went no further. The initial idea (this was 2013!) was to rewrite the project in Ruby: I had been using Ruby at work and it looked like a decent replacement for Perl. My main problem was the difficulty of installation in an uncontrolled environment. If you have control over your environment (it's your laptop or you are in charge of the server configuration via Puppet or similar) then the task is easy. But if you ever need to deploy somewhere with little or no notice, it becomes a problem: there are servers where Perl is not installed, and is common that the server also have a policy forbidding all scripting languages from being deployed. …

Uber Engineering

Uber’s mission is transportation as reliable as running water, everywhere, for everyone. Last time, we talked about the foundation that powers Uber Engineering. Now, we’ll explore the parts of the stack that face riders and drivers, starting …

The post The Uber Engineering Tech Stack, Part II: The Edge and Beyond appeared first on Uber Engineering Blog.

I was on the quest of searching the Holy Grail of Go programming, and I found something, which I doubt that it is, but close enough – for the first sight.

I have several problems with GO, first, that I write my code on an OSX box, and I’ll run the programs on Linux hosts, so I have to solve the cross compilation; my second problem with Go, that I don’t really like the “There is a GO project folder, and all the GO projects are relying on” approach. It makes using GitHub painful.

The first problem of mine is easily achievable since GO 1.5: we only need a GOOS environment variable and we can compile to different OS-es (see more at Dave Cheney: http://dave.cheney.net/2015/08/22/cross-compilation-with-go-1-5) easily.

The second problem is easily solvable too, just we have to start using the GOPATH variable for every GO project we have.

I don’t really want to use any external dependencies, so I decided to …

A few years ago, I wrote the original OmniSQL as a Perl script to automate the execution of queries across many MySQL servers and, in some cases, provide basic aggregation functionality. Shortly after that time, Justin Swanhart released Shard-Query and Domas Mituzas released pmysql, so OmniSQL was retired.

I’m a daily user of pmysql; indeed, we run a heavily patched version. Lately, I find myself needing to extend the functionality even more than we’ve already done. After a bit of evaluation, I decided that C wasn’t the best language to move forward with the new features. Thus, OmniSQL is reborn in Go. While initial testing indicates that it is an order of magnitude slower than pmysql running against 1,000+ instances of MySQL, it’s still well within my tolerance (and there are many areas of …

With MySQL, it is relatively easy to create “point in time” restores. All you need is recent(ish) backup and a bunch of saved binary logs. You can restore the backup you have, and when it is completed, you can use mysqlbinlog utility to apply your saved binary logs to the desired state of your database.

I have created a simple go application to make your life easier. You can find it on my GitHub page.

The app works as reading its config file for the MySQL server connection credentials, a local directory where the binary logs will be kept, and the will path of the mysqlbinlog utility.

1. It checks the binlogs on the remote server which able to streamed
2. Checks the local directory where the binlogs are kept, to check which logs are already there
3. The incomplete (file size differs local and on remote server) files will be …

With MySQL it is relatively easy to create point in time restores. All you need is a recent(ish) backup, and a bunch of saved binary logs. You can restore the backup you have, and when it is completed, you can use mysqlbinlog utility to apply your saved binary logs to the desired state of your database. I […]