What You May Not Know About Random Number Generation in Sysbench

Sysbench is a well known and largely used tool to perform benchmarking. Originally written by Peter Zaitsev in early 2000, it has become a de facto standard when performing testing and benchmarking. Nowadays it is maintained by Alexey Kopytov and can be found in Github at https://github.com/akopytov/sysbench.

What I have noticed though, is that while widely-used, some aspects of sysbench are not really familiar to many. For instance, the easy way to expand/modify the MySQL tests is using the lua extension, or the embedded way it handles the random number generation. 

Why This Article? 

I wrote this article with the intent to show how easy it can be to customize sysbench to make it what you need. There are many different ways to extend sysbench use, and one of these is …

One problem I have experienced during logical exporting and importing databases is the tools don’t output any progress indicator (or anything at all really) and using the -v (verbose) switch outputs every single MySQL command being run from the import file. As, importing and exporting of databases  is one of the most frequent activity being performed , I Just felt the need of some tool that could help me check the approximate progress of import and export of databases going on.

So, I searched and fortunately could find a tool named Pipeviewer (PV) which allows a user to see the progress of data through a pipeline, by giving information such as time elapsed, percentage completed (with progress bar), current throughput rate, total data transferred, and ETA .

Installation :

On Centos and RHEL :

[root@vm1 vagrant]# yum install pv

[root@vm1 …

In the MySQL ecosystem, orchestrator is the most popular and well-respected high availability and topology management tool, integrating well with other solutions such as ProxySQL. It facilitates automatic (or manual) discovery, refactoring and recovery of a replicated MySQL environment, and comes complete with both command-line (CLI) and web interfaces for both humans and machines to interact with.

As we all know, humans are prone to errors and as such accidents can happen, particularly when humans and computers interact with each other! Recently, one of these situations related to the web interface of orchestrator during topology refactoring with its drag-and-drop capabilities, where a drop occurred unintentionally and thus had an impact on replication.

When …

Recently one of our client approach Mydbops with Query slowness on a MySQL environment . They deployed the new code for generate the huge reports for the year end analytics data . After the deployment the queries were extremely slow and they struggled lot , then they approached us for the solution. After the analysis, their OLAP database as expected it was IO bound with 100% disk IOPS utilised during the report generation. So, the queries were starving for the Disk IO slows the process .

Problem statement :

• Reports are majorly focused on two larger log tables ( emp_Report_model , emp_details ) .
• The report generator (procedure) is using the count(*) statement to stimulate the aggregated data on each call. It is required for their business purpose .
• Count(*) is terribly slow in MySQL ( Using MySQL 5.7 ) as it …

In this post, I am going to show you how to run Orchestrator on FreeBSD. The instructions have been tested in FreeBSD 11.3 but the general steps should apply to other versions as well.

At the time of this writing, Orchestrator doesn’t provide FreeBSD binaries, so we will need to compile it.

Preparing the Environment

The first step is to install the prerequisites. Let’s start by installing git:

[vagrant@freebsd ~]$ sudo pkg update
Updating FreeBSD repository catalogue...
Fetching meta.txz: 100% 944 B 0.9kB/s 00:01
Fetching packagesite.txz: 100% 6 MiB 492.3kB/s 00:13
Processing entries: 100%
FreeBSD repository update completed. 31526 packages processed.
All repositories are up to date.

[vagrant@freebsd ~]$ sudo pkg install git
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
All repositories are up to date.
New …

In the past year, GitHub engineers shipped GitHub Packages, Actions, Sponsors, Mobile, security advisories and updates, notifications, code navigation, and more. Needless to say, the development pace at GitHub is accelerated.

With MySQL serving our backends, updating code requires changes to the underlying database schema. New features may require new tables, columns, changes to existing columns or indexes, dropping unused tables, and so on. On average, we have two schema migrations running daily on our production servers. Some days we have a half dozen migrations to run. We’ll cover how this amounted to a significant toil on the database infrastructure team, and how we searched for a solution to automate the manual parts of the process.

What’s in a migration?

At first …

In MySQL 8, Information Schema was basically re-written to utilize the all-New Data Dictionary which made it faster and better in many ways. Yet it also introduces some very counterintuitive behaviors when it comes to how actual and consistent data is.

Here is a simple test.  Let’s create a table and insert a bunch of rows in it… and then check table size:

mysql> show table status \G
*************************** 1. row ***************************
           Name: t
         Engine: InnoDB
        Version: 10
     Row_format: Dynamic
           Rows: 0
 Avg_row_length: 0
    Data_length: 16384
Max_data_length: 0
   Index_length: 0
      Data_free: 0
 Auto_increment: NULL
    Create_time: 2019-12-27 01:04:37
    Update_time: NULL
     Check_time: NULL
      Collation: utf8mb4_0900_ai_ci
       Checksum: NULL
 Create_options:
        Comment:
1 row in set (0.02 sec)

mysql> select count(*) from t;
+----------+
| count(*) |
+----------+
| …

For Linux, the most common way to distribute software is binary packages in the rpm or deb format. Most packages are included in the official distribution repositories or 3rd party software repositories. Nevertheless, there are some cases where you need to install just a few standalone packages.   You might be able to use the local package install tools, namely dpkg or rpm, however, there are cases where packages can’t be installed due to the dependencies and you need to install all dependencies manually. It might take some time and isn’t always an easy process. But there is a solution that can help – you can create your own local repository and deploy your packages to it.

Let’s discuss how to create your local repositories to make your life easier.

RPM-Based Distributions

RPM-based operating systems work with rpm packages and the most common package manager for them is yum.   While newer …

When you have multiple database servers working together as nodes in a cluster, it’s beneficial to understand how data consistency is established. In this post, we’re going to take a look at the various cluster consistency levels that you can choose from within InnoDB Group Replication and see how they impact data flow and failure recovery.

Let’s start with a quick refresher on what data consistency is. In a nutshell, consistency is just a way of thinking about how data is duplicated across multiple nodes in a cluster. If you write a row of data to one node, that data is not considered to be consistent within the cluster until it has been written to all of the other participating nodes. Without consistency, it’s possible to write data to one node in the cluster but not see it immediately when reading from another node in the cluster. In some cases, the aforementioned scenario, better known as eventual consistency, is acceptable but …

In this post, we will explore one approach to MySQL high availability with ProxySQL, Consul and Orchestrator.

This is a follow up to my previous post about a similar architecture but using HAProxy instead. I’ve re-used some of the content from that post so that you don’t have to go read through that one, and have everything you need in here.

Let’s briefly go over each piece of the puzzle:

– ProxySQL is in charge of connecting the application to the appropriate backend (reader or writer).

It can be installed on each application server directly or we can have an intermediate connection layer with one or more ProxySQL servers. The former probably makes sense if you have a small number of application servers; as the number grows, the latter option becomes more attractive. Another scenario for the …