I recently became aware of WeSQL. A MySQL-compatible database that separates compute and storage, using S3 as the storage layer. The product uses a columnar format by default which is significantly more space-efficient than InnoDB.

WeSQL introduces a new storage engine called SmartEngine using a LSM-tree-based structure that is ideal for a storage bucket implementation, and documentation shows the implementation of raft replication to combat latency concerns. There is a lot more information to review, the serverless architecture and WeScale, a database proxy and resource manager.

It was very easy to take it for an initial spin using a docker container and an AWS S3 bucket. I would really like to try CloudFlare R2 which implements the S3 API.

Under the covers there …

A recommendation we often give to our customers is along the lines of “archive old data” to reduce your database size. There is a tradeoff between keeping all our data online and archiving part of it to cold storage.

There could also be legal requirements to keep certain data online, or you might want to query old data occasionally without having to go through the hassle of restoring an old backup.

In this post, we will explore a very useful feature of AWS RDS/Aurora that allows us to export data to an S3 bucket and run SQL queries directly against it.

Archiving Data to S3

Let’s start by describing the steps we need to take to put our data into an S3 bucket in the required format, which is called Apache Parquet.

Amazon states the Parquet format is up to 2x faster to export and consumes up to 6x less storage in S3, compared to other text formats.

1. Create a snapshot of the database (or …

Cloud backup is a strategy increasingly used in organizations. Send copies of your data to the cloud, can help you to prevent a devastating IT crisis and ensure business continuity.

Currently, MySQL Enterprise Backup supports the following types of cloud storage services:

- Oracle Cloud Infrastructure (OCI) Object Storage

- OpenStack Swift or compatible object storage services

- Amazon Simple Storage Service (S3) or compatible storage service

The post Create a Cloud Backup with MySQL Enterprise Backup first appeared on dasini.net - Diary of a MySQL experts.

We are living in the DataLake world. Now almost every oraganization wants their reporting in Near Real Time. Kafka is of the best streaming platform for realtime reporting. Based on the Kafka connector, RedHat designed the Debezium which is an OpenSource product and high recommended for real time CDC from transnational databases. I referred many blogs to setup this cluster. But I found just basic installation steps. So I setup this cluster for AWS with Production grade and publishing this blog.

A shot intro:

Debezium is a set of distributed services to capture changes in your databases so that your applications can see those changes and respond to them. Debezium records all row-level changes within each database table in a change event stream, and applications simply read these streams to see the change events in the same order in which they occurred.

Basic Tech Terms:

• Kafka …

We are living in the DataLake world. Now almost every organizations wants their reporting in Near Real Time. Kafka is of the best streaming platform for realtime reporting. Based on the Kafka connector, RedHat designed the Debezium which is an OpenSource product and high recommended for real time CDC from transnational databases. I referred many blogs to setup this cluster. But I found just basic installation steps. So I setup this cluster for AWS with Production grade and publishing this blog.

A shot intro:

Debezium is a set of distributed services to capture changes in your databases so that your applications can see those changes and respond to them. Debezium records all row-level changes within each database table in a change event stream, and applications simply read these streams to see the change events in the same order in which they occurred.

Basic Tech Terms:

• Kafka …

OblakSoft is pleased to announce general availability of the Cloud Storage Engine for MySQL (ClouSE) version 1.0.  Whether you want to use ready-to-run WordPress for Amazon Cloud / Google Cloud, configure your own WordPress installation with WP2Cloud plugin, or use ClouSE to deploy your own applications, ClouSE lets you tap the power of cloud storage to store your data extremely durably and make serving the content highly scalable.

We would like to thank more than 2,000 registered Beta users and early adopters who helped …

MySQL Enterprise Backup(MEB) has been widely used as the most efficient tool to take backups of huge databases. The storage of backups can be done on-premises – viz, local hard disk, external hard disk drives, network mounted disk. MEB can also stream backups to tape by supporting different Media Management Software like Symantec NetBackup, EMC2 and Oracle Secure Backup. Streaming of backups can also be done using 'ssh' to another machine.

As data keeps growing, instead of expanding the storage, database users are choosing to rely on the growing trend which is the Cloud Storage.

A few facts about Cloud Storage:

1. Offers off-premises storage with AAA(Anytime, Anywhere Access)
2. Sharing of data to make a collaborative effort.
3. Highly fault tolerant, durable and scalable and secure.
4. Cost-effective

To the benefit of our MySQL database users, Mysql …

Shard-Query (source) now supports the MySQL “LOAD DATA INFILE” command.

When you use LOAD DATA LOCAL INFILE a single threaded load from the current process will be performed.  You can specify a path to a file anywhere readable by the PHP script.  This allows loading without using the Gearman workers and without using a shared filesystem.

If you do not specify LOCAL, then the Gearman based loader is used.  You must not specify a path to the file when you omit the LOCAL keyword.  This is because the shared path will the pre-pended to the filename automatically.  The shared path must be a shared or network filesystem (NFS,CIFS,etc) and the files to be loaded must be placed on the shared filesystem for the Gearman based loader to work.  This is because workers may run on multiple nodes and …

ClouSE 1.0b.1.8 is released

OblakSoft is pleased to announce the release of ClouSE Beta version 1.0b.1.8.  This is a minor maintenance release that addresses a couple pain points, while we’re working on a release candidate that will bring the built-in point-in-time recovery and other major features.

Here is the summary of changes:

• Improved caching for Weblobs

Now ClouSE sets the cache control headers for Weblobs.  The cache control headers specify the max-age directive that allows caching content for one year.  This makes proxies and content delivery networks (CDN) more efficient.  The change resolves this support issue.

• Improved error handling on startup

Now database requests fail if …

Updating ClouSE can now be done with one command

With more than a thousand registered users worldwide, guiding customers through the upgrade process is getting hard, so we decided to automate the servicing steps.  Now the update-clouse script can be used to update ClouSE binaries to the current version.  This script ships with the ClouSE binary distribution.

Here is an example of updating ClouSE using the Webmin console.

First, download ClouSE 64-bit binaries from http://www.oblaksoft.com/downloads/.  Then upload and extract the binaries to the /home/ec2-user directory. In Webmin, navigate to Others, then to Upload and Download, then …