Home > Programming, Python > How to copy a file really slowly using Python

How to copy a file really slowly using Python

Just realized that the poor performance I was seeing when downloading a file from Rackspace Cloudfiles to a local drive was due to shutil.copyfileobj without specifying a buffer size.  As I found out, the default buffer is 16k (just in case you’re running on a 640k server), increasing it to 1MB sped up the process from 17.8 seconds to 0.8 seconds, with 5MB down to 0.3 seconds (for files averaging about 3-4MB).

So remember: always supply a buffer size, and the 60x speed increase can be yours too!

Advertisements
Categories: Programming, Python
  1. No comments yet.
  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: