0 2 * * * echo test > test.`date +%Y-%m-%d`

This will create a file called “test.“, that will be overwritten every day …and here is why (from man crontab):

“Percent-signs (%) in the command, unless escaped with backslash (\), will be changed into newline characters, and all data after the first % will be sent to the command as standard input.”

This works…

0 2 * * * echo test > test.`date +\%Y-\%m-\%d`

no: command not found

What is the “no” command and why and who tries to call it!?

Well, what happened was: I though I was being clever and used RSYNC=$(which rsync) to find the rsync command to execute. Unfortunately rsync actually is located in /usr/local/bin on this system and /usr/local/bin is not in the PATH, so which could not find it. Running the script from a shell worked, because that user just happened to have /usr/local/bin appended to it’s PATH environment variable. The “no” comes from the message which returns when it cannot find the command:

no rsync in /usr/bin /sbin /usr/sbin /bin /usr/bin

I am not sure how this is supposed to work though. From Amazon you get a private and public key after signing up for an AWS account. You use the public key in plain text as a parameter in your request. In addition you now (since August 2009 as it seems) have to add a timestamp and compute a signature key using the whole request string (including that timestamp and your public key) and the private key. Now append that signature as another parameter to the request and send it.

Of course every request will have a different signature because (a) you want to query for different stuff and (b) time changes and the timestamp must be recent or Amazon will consider the request as “old” and not worthy of an answer.

So what do you do now? Include your private key in your Python source, that you put out in a public repository? You probably shouldn’t, but that means the customer (user?) must have it’s own public/private key pair from Amazon. What a major inconvenience for the customer, to be required to have a AWS account just to get some covers.

Anyway, back to Listen. I created a branch of the 0.6 series on Launchpad  (a first for me, still getting used to bzr) and committed the necessary changes to amazon.py in revision 1126. Today I merged changes from the parent 0.6 and therefore that file now contains both public and private key — of someone else, not mine.

By the way, he original amazon.py is from of PyAWS. Unfortunately a later release of PyAWS 0.3 was not uploaded to SourceForge and further changes by others for the signature stuff also never found their way back to the repository. Isn’t open source great? </sarcasm>

When inserting a lot of data it is better to create one huge insert statement then thousands of small ones. When you do that you have to make sure that your server accepts huge amounts of data by setting an appropriate value for max_allowed_packet.

Furthermore the MySQL protocol defines a “MySQL packet” to have a header of three bytes for the length followed by a one byte counter. The packet body starts with one byte command followed by data (a SQL query or response). With three bytes you can count up to around 16 million or in other words a MySQL packet can only be 16MB long (plus four header bytes), because that is the maximum value for the length field.

What Net::MySQL does in this case is: nothing. It sets a length value that is too big for three bytes, ignores the need for a counter and just keeps pouring data into the socket like there is no tomorrow without splitting into packets. The server reacts with a connection reset and “packets out of order” error. At least that is what we have observed.

I’ve sent a bug report and patch to Net::MySQL’s bug tracker.