I’ve been playing around with some quick system automation scripts that are handy to use when you don’t want / need to setup a chef or puppet action. I like to keep all of my hostnames and login details in a MySQL database (a cmdb actually) but for this example we’ll just use a couple of nested lists. This script executes commands in parallel across the hosts you choose in the menu system via the “pdsh” command, so make sure you have that installed before running. Alternately you can change the command call to use ssh instead of pdsh for a serialized execution, but that’s not as fun or fast. With some customizations here and there you can expand this to operate parallelized jobs for simplifying daily work in database administration, usage reporting, log file parsing, or other system automation as you see fit. Here’s the code. Comments welcome as always!

#!/usr/bin/env python
## NAME: menu_parallel_execution.py
## DATE: …

In my previous post I described what it took to add SQL support and a simple command line client to a NoSQL storage. However, I needed not just ad-hoc testing with a client, but a framework to automatically run and manage many tests.
I expect that automated tests are easy to understand, extend, and maintain. When a test breaks, finding and debugging what broke should be easy. Such qualities can not be met in a heterogeneous test environment. Rather, it would be best if some common language and toolkit was used. It's easiest for all when a failing test can be run directly under a debugger.
In MySQL, this task is solved with a combination of 'mysqltest' client-side testing tool and 'mysql-test-run', an automation environment for functional tests.
'mysqltest' is …

Packt Publishing recently sent me a copy of MySQL for Python to review and after reading through the book I must say that I’m rather impressed at the variety of topics that the book covers.

It starts off with the basics of setting up MySQL for your testing/development needs by going over several of the common installation and configuration methods. After that it’s a quick intro for connection methods and simple error reporting for connections. The author gives a quick intro to CRUD and how it relates to databases and python before heading into the common tasks of simple queries. I was surprised to see some database profiling discussion; which is rather handy for a new coder or a person new to MySQL. Once the basics of …

Every so often you need to use a queue to manage operations in an application. Python makes this very simple. Python also, as I’ve written about before, makes threading very easy to work with. So in this quick program I’ll describe via comments, how to make a simple queue where each job is processed by a thread. Integrating this code to read jobs from a mysql database would be trivial as well; simply replace the “jobs = [..." code with a database call to a row select query.

#!/usr/bin/env python
## DATE: 2011-01-20
## FILE: queue.py
## AUTHOR: Matt Reid
## WEBSITE: http://themattreid.com
from Queue import *
from threading import Thread, Lock

'''this function will process the items in the queue, in serial'''
def processor():
    if queue.empty() == True:
        print "the Queue is empty!"
        sys.exit(1)
    try:
        job = queue.get()
        print "I'm operating on job item: %s"%(job)
        queue.task_done()
    except:
        print …

Tarantool offers clients a simple binary protocol with basic data manipulation commands - GET, PUT, SET, DELETE. All administrative commands, however, must be sent in textual form to a separate, administative port. A separate port is useful as long as there is no authentication support. Examples of administrative commands are 'SAVE SNAPSHOT' or 'SHOW STAT'.
The whole thing had a few functional tests, but all of them required to be run manually. If you're not the one who wrote it, you probably wouldn't know how to run it.
I was looking for something that would be easy to write and easy to run. It needed to be a lingua franca of testing, used both by developers and quality assurance engineers. It would also be nice to be able to easily express in the new framework test cases for discovered bugs.
Of course, for an ex-SQL geek, SQL looked very much like the lingua …

MySQL Connector/Python 0.3.2, a development release, is available for download:
https://launchpad.net/myconnpy/+download

Disclaimer: Since version 0.3 is still a development release, or ‘alpha’, it is not
recommended to run this in production.

MySQL Connector/Python 0.3.2-devel is a maintenance release fixing following bugs:

• lp:701081 -Doesn’t install with Python 2.4

About MySQL Connector/Python: MySQL Connector/Python is implementing the
MySQL Client/Server protocol completely in Python. No MySQL libraries
are needed, and no compilation is …

MySQL Connector/Python 0.3.1, a development release, is available for download:
https://launchpad.net/myconnpy/+download

Disclaimer: Since version 0.3.1 is still a development release, or ‘alpha’, it is not
recommended to run this in production.

MySQL Connector/Python 0.3.1-devel is a maintenance release fixing following bugs:

• lp:695514 – Infinite recursion when setting connection client_flags
• lp:691836 – Incorrect substitution by cursor.execute when tuple args contains ‘%s’

…

Setting client flags with MySQL Connector/Python works a bit differently than the other MySQL Python drivers. This blog post describes how to set and unset flags, like the CLIENT_FOUND_ROWS.

The default client flags for the MySQL Client/Server protocol can be retrieved using the constants.ClientFlag class:

>>> from mysql.connector.constants import ClientFlag
>>> defaults = ClientFlag.get_default()
>>> print ClientFlag.get_bit_info(defaults)
['SECURE_CONNECTION', 'TRANSACTIONS', 'CONNECT_WITH_DB',
 'PROTOCOL_41', 'LONG_FLAG', 'MULTI_RESULTS',
 'MULTI_STATEMENTS', 'LONG_PASSWD']

To set an extra flag when connecting to MySQL you use the client_flags argument of connect()-method. For example, you’d like to have the …

MySQL v5.5 is GA, but is it working with Python? Yes, it does. Below you’ll find some quick, small tests I did with MySQLdb, oursql and our own MySQL Connector/Python.

My desktop is a Mac, but when it works on that, I’m sure it works elsewhere too. If not, just let us know!

MySQL for Python (aka MySQLdb)

Installing MySQL v5.5.8 64-bit from tar ball on MacOS X 10.6, it compiled fine and the module loaded giving me the …

MySQL Connector/Python 0.3.0, a development release, is available for download:
https://launchpad.net/myconnpy/+download

Since version 0.3.0 is still a development release, or ‘alpha’, it is not
recommended to run this in production.

MySQL Connector/Python 0.3.0 adds following features:

• Python v2.4 support is back.
• Support for compressed protocol.
• Support for SSL connections (when Python’s ssl module is available).
• Support for packets which are bigger than 16MB.
• Max allowed packetsize defaults to 1GB.
• Some performance improvements.

See the ChangeLog for extra details.

Please …