An rsync-like utility for Amazon S3 and Google Storage

by on Dec.11, 2011, under F/OSS, Linux, Python

Every so often I find myself working on that odd job that requires syncing files with Amazon’s S3. In the beginning, I tried some of the various S3 FUSE interfaces—hoping for something that would play nice with rsync—but FUSE’s stability always left something to be desired and more often than not I’d be left with that one transfer that never would quite finish correctly.

Eventually I discovered boto and settled in to using a hacked together (but stable) Python/boto solution for these type of tasks—all the while wondering why nobody took the time to write a “real” rsync-like client for S3.

Well, this last time around I finally decided to stop whining and take matters into my own hands. After a couple of late nights fleshing out my original boto solution, I’m happy to announce what I’m calling “boto rsync”—an rsync like wrapper for boto’s cloud storage interfaces (both S3 and Google Storage).

Please take a look at the project on github and let me know what you think:

:, , , ,

  • rainbard

    Tried it out and it worked well for me. Thanks. Is it possible to just copy file references and not the entire file. That is, create a zero byte file on my local file system if the corresponding full file exists on S3?

  • Horacio Degiorgi

    how can i use this with DreamObjects  ?
    gs:/ for google
    s3:/ for amazon
    what i need for DreamObjects ?

  • Eugê

    You just need to specify an endpoint: –endpoint

    Otherwise you can use the s3:// prefix.

    For example my command line looks like this:

    “path-topython.exe” “” -a YOUR-ACCESS-KEY -s
    YOUR-SECRET-KEY –endpoint –delete “h:/Photo/”

    I hope I could help you.

  • papa-john

    Any hard data on performance?

    i am trying to sync 6+ mil files between 2 buckets.

    What about at least some basic atribute caching to speed further runs (is it even possible?)

Looking for something?

Use the form below to search the site:

Still not finding what you're looking for? Drop a comment on a post or contact us so we can take care of it!

Visit our friends!

A few highly recommended friends...