The ability to import data in parallel is now a feature of
the new MySQL Shell. And you can run it from a command line
instead of from within an interactive shell.
How Does It Work?
The importTable utility, new on MySQL Shell 8.0.17,
analyzes the input file, divides that file into chunks, and then
uploads that data using parallel connections. This is much
faster than the LOAD DATA statement. You get the ability to
define how the data is delimited as far a field and lines are
defined. And it works on DOS CSVs, Unix CSVs, TSVs, and
JSON if that JSON is in one document per line mode. You also
can adjust the number of threads, number of bytes sent per each
chunk, and the maximum rate of data transfer per thread so you
can balance the load on the network and the speed of data …
As I mentioned on my last post, where I compared the default configurations options in 5.6 and 5.7, I have been doing some testing for a particular load in several versions of MySQL. What I have been checking is different ways to load a CSV file (the same file I used for testing the compression tools) into MySQL. For those seasoned MySQL DBAs and programmers, you probably know the answer, so you can jump over to my 5.6 versus 5.7 results. However, the first part of this post is dedicated for developers and MySQL beginners that want to know the answer to the title question, in a step-by-step fashion. I must say I also learned something, as I under- and over-estimated some of the effects of certain …
[Read more]On my post last week, I analysed some of the most common compression tools and formats, and its compression speed and ratio. While that could give us a good idea of the performance of those tools, the analysis would be incomplete without researching the decompression. This is particularly true for database backups as, for those cases where the compression process is performed outside of the production boxes, you may not care too much about compression times. In that case, even if it is relatively slow, it will not affect the performance of your MySQL server (or whatever you are using). The decompression time, however, can be critical, as it may influence in many cases the MTTR of your whole system.
Testing …
[Read more]