Just realized that the poor performance I was seeing when downloading a file from Rackspace Cloudfiles to a local drive was due to shutil.copyfileobj without specifying a buffer size. As I found out, the default buffer is 16k (just in case you’re running on a 640k server), increasing it to 1MB sped up the process from 17.8 seconds to 0.8 seconds, with 5MB down to 0.3 seconds (for files averaging about 3-4MB).
So remember: always supply a buffer size, and the 60x speed increase can be yours too!
Over the past couple of days I attended several OSCON sessions on new (to me) programming languages:
- Erlang was not completely new to me, but I had not understood the overall architecture of the Erlang environment and its similarities to an operating systen.
- Clojure, a Lisp dialect, seems the least practical for day-to-day use, but is also the most different and interesting to me.
Since a new “hardest” Sudoku puzzle was just announced, I finally wrote my own Sudoku solver in Python. I based it on some ideas from Peter Norvig’s Sudoku solver, but ended up writing everything from scratch. The code is available on github.