Backup with duplicity on Google Cloud Storage

Question

I have already set a backup routine that backups all my files to Amazon S3 using duplicity. I would like to create a similar backup routine with Google Cloud Storage. Is it possible to get duplicity to work with GCS?

Answer 1

There is now (since 2013-07-19 - version 0.6.22) support to use Google Cloud Storage directly in duplicity.

You should use gsutil config -a to create a ~/.boto config file. And then you can use duplicity like this:

duplicity local_dir gs://yourbucket

Also of note is that only the old authentication mechanism is supported. You should enable "Interoperable Access" from your Google Storage Console.

Detailed steps on http://www.willdurness.com/back-up-your-vm-to-google-cloud-storage-using-duplicity.

Answer 2

• As of August 2012, Duplicity does not support Google Cloud Storage (GCS); a search of the duplicity-talk mailing list archives shows that there has been no discussion, even at a wishlist stage, of adding such support.
  • Google Docs is supported, but that API is totally different from GCS.
  • Wishlist bug filed recently
• While the Python boto module used as a back-end by Duplicity to provide S3 support also supports GCS, adding GCS support to duplicity would require non-trivial modifications to its code; the S3 and GCS APIs are not identical.
  • If anyone wants to give adding GCS support a go, it would be simplest to start with and modify the _boto_single.py file which provides the backend for single-threaded S3 transactions.
• That GCS isn't widely supported shouldn't come as a surprise. It is nowhere near as universal as S3; for example, you must register as a developer with Google and the service is still considered part of the "beta" Google Labs.
  • The only backup solution I could find supporting GCS is Zmanda (commercial).
  • As Google rolls out GCS on a wider basis--beyond just developers--I think we should start seeing support for it in FOSS apps.
• For now, to use Duplicity with GCS, you must either backup to S3 and then use Google's gsutil tool to sync S3 with GCS, as Jorge's answer indicates; alternately, you could use Duplicity to backup locally and then sync that with GCS using gsutil.

Answer 3

This is a partial-answer that might help the next person:

So far it seems that copying things to Google Drive/Docs is supported:

duplicity localdir gdocs://username:password@gmail.com/remotedir/

However this is different from Google Cloud Storage. I think this should be doable since duplicity and gsutil both use boto, but I'm not sure if this requires work to be done in duplicity itself though.

As a work around until you sort this, you can certainly copy directly from the s3 to the GCS buckets:

gsutil -m cp -R s3://yourbucket gs://yourbucket

Answer 4

Version v0.6.22 (2013/08/22) of Duplicity added Google Cloud Storage support.

Answer 5

deja-dup 34.1 (duplicity's GUI) now has "experimental support for Google Cloud Storage" (https://launchpad.net/deja-dup/+milestone/34.1)

(I tried enabling interoperability for HMAC authentication on my GCS buckets and specifying an S3 Bucket in earlier versions of Deja-Dup - similar to how duplicity does it - but to no avail)