In this blog, I will share the steps I took to debug an error ‘ERROR 1030 (HY000): Got error 168 – ‘Unknown (generic) error from engine’ from storage engine’ while…

The post How to fix 1030 Unknown generic error from engine in MySQL first appeared on Change Is Inevitable.

There are plenty of articles on the Internet that tell you to setup a Galera Cluster by disabling an OS based firewall and also disabling SELinux. While we agree that this might be the fastest way to get your Galera Cluster setup, it is not necessarily good security hygiene, and we would prefer if you started 2022 with a bit more secure Galera Cluster!

What is SELinux? Is is Security-Enhanced Linux that allows administrators to allow who has more control over the system. It has permissive and enforcing mode, and is turned on by default in Red Hat Enterprise Linux and derivatives. It is important to remember that if you install Galera Cluster via a package that we provide, we have provided all the necessary contexts for it. You effectively do not have to disable SELinux to get started.

However, if you are using the rsync method for a …

Disable SELinux? Think again!

SELinux is always a complicated topic. If you search this on the web, most people will advise just disabling it, but that may not be acceptable from a security point of view in your organization. In this case, we are going to see how to solve an issue with SELinux and MySQL log rotation

We had configured log rotation as per this post. The scripts seemed to work perfectly when running manually. However, when running under cron they would fail to run. Since there were no other errors in the logs, eventually I tracked that down to SELinux. What I found is that SELinux default policies prevent logrotate daemon from making the changes to files outside of /var/log. In this case, MySQL logs were living on /var/lib/mysql so that was clearly the problem.

Figuring out SELinux

The first thing to do when debugging a …

Why do I spend time blogging about security frameworks? Because, although there are some resources available on the Web, none apply to Percona XtraDB Cluster (PXC) directly. Actually, I rarely encounter a MySQL setup where SELinux is enforced and never when Percona XtraDB Cluster (PXC) or another Galera replication implementation is used. As we’ll see, there are good reasons for that. I originally thought this post would be a simple “how to” but it ended up with a push request to modify the SST script and a few other surprises.

Some context

These days, with all the major security breaches of the last few years, the importance of security in IT cannot be highlighted enough. For that reason, …

In this blog post, I’ll look at how to make Percona XtraDB Cluster and SELinux work when used together.

Recently, I encountered an issue with Percona XtraDB Cluster startup. We tried to setup a three-node cluster using Percona XtraDB Cluster with a Vagrant CentOS box, but somehow node2 was not starting. I did not get enough information to debug the issue in the donor/joiner error log. I got only the following error message:

2018-02-08 16:58:48 7910 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'joiner' --address '192.168.100.20' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '7910' --binlog 'mysql-bin' '
2018-02-08 16:58:48 7910 [ERROR] WSREP: Failed to …

I recently worked with a customer who had a weird issue: when their MySQL server was started (Percona Server 5.5), if they try to run service mysql start a second time, the init script was not able to detect that an instance was already running. As a result, it tried to start a second instance with the same settings as the first one. Of course this fails and this creates a mess. What was the issue? A missing rule in SELinux. At least it looks like

Summary

If SELinux is set to enforcing and if you are using Percona Server on CentOS/RHEL 6 (other versions could be affected), service mysql start doesn’t work properly and a fix is simple to run:

# grep mysqld_safe /var/log/audit/audit.log | audit2allow -M mysqld_safe
# semodule -i mysqld_safe.pp
# service mysql restart

Other options are:

• Set SELinux to permissive
• Use the CentOS/RHEL standard MySQL init script …

MySQL ERROR 1018 (HY000): Can't read dir of './sakila/' (errno: 13 - Permission denied) caused due to moving database to a different partition and using softlink on CentOS and it's fix for SELinux.

The post MySQL Permission denied error 13 and solution first appeared on Change Is Inevitable.

If you're setting up MySQL Cluster on Oracle Linux or another Linux such as Red Hat, CentOS or Fedora, you might have come across some problems getting the nodes to communicate. There are two ways you might bump into problems with network connectivity: The iptables firewall, and SELinux. These security mechanisms might prevent your various nodes—management, data, and API—from communicating with each other in various ways and with various symptoms.

Let's have a look at what you're likely to see. 

Data nodes stuck at "starting"

The first thing you might notice is that your data nodes get stuck in the "starting" state. Running show in the management client gives something like this:

[ndbd(NDB)]     2 node(s)
id=2    @192.168.56.211  (mysql-5.6.11 ndb-7.3.2, Nodegroup: 0, Master)
id=3    @192.168.56.212  (mysql-5.6.11 ndb-7.3.2, starting, Nodegroup: 0)

...and it never moves away from starting.

…

I've previously written about AppArmor and MySQL, and how to change MySQL's default file locations on systems with AppArmor enabled. Ubuntu and SUSE ship with AppArmor enabled, but some other distributions such as Oracle Linux don't, along with related distrubutions such as Red Hat, CentOS and Fedora. Rather, these other distributions use another mandatory access control system called SELinux. 

Here's some technical detail that might come in handy later.

SELinux uses concepts such as types and domains. Types belong to resources such as files and ports; these are the "objects" in SELinux. Domains contain the "subjects" (processes) and object types that are associated with each other in some way, for example because they are all related to …

There was one MySQL server with a Adaptec Raid controller and 4 disks. One of the disks was having media errors and caused the whole SCSI bus to become unavailable.

This resulted in a corrupted InnoDB table.

Luckily we did have backups. A full backup and incrementals.

So to restore the backups I installed XtraBackup and MySQL 5.5 on another server.

Then the first step was to 'prepare' the backup. This worked okay for the full backup (redo only).

The second step to add the incremantals failed for the first incremental. This was easily resolved by specifying the full paths instead of relative paths.

Then the backup was fully prepared using the redo logs and undo logs.

As XtraBackup doesn't backup your my.cnf we copied the my.cnf from another server and adjusted it for this server. The my.cnf in your backup only contains everything needed for a …