1

I am trying to extract a backup from an old duplicity back up (done by the ubuntu gui, I believe it was deja-dup).

I copied all I could off a dying hard drive. I have

  • a lot of difftar.gz files - some 2014 and some 2016.
  • a single .manifest file from 2014.
  • no sigtar.gz files

The backups are not encrypted - I have unzipped one difftar.gz file and examined it in a hex editor and I can see lots of plain text.

Running some commands:

% duplicity --no-encryption --ignore-errors collection-status file:///home/hamish/poonbackup2014
Running in 'ignore errors' mode due to --ignore-errors; please re-consider if this was not intended
Last full backup date: Mon Sep 15 19:37:44 2014
Collection Status
-----------------
Connecting with backend: BackendWrapper
Archive directory: /home/hamish/.cache/duplicity/c34b5c3ae7d763a715fd062ec5f49baa

Found 1 secondary backup chain.
Secondary chain 1 of 1:
-------------------------
Chain start time: Mon Sep 15 19:37:44 2014
Chain end time: Mon Sep 15 19:37:44 2014
Number of contained backup sets: 1
Total number of contained volumes: 2063
 Type of backup set:                            Time:   Number of volumes:
                Full         Mon Sep 15 19:37:44 2014              2063
-------------------------

No backup chains with active signatures found
No orphaned or incomplete backup sets found.



% duplicity --no-encryption --ignore-errors collection-status file:///home/hamish/poonbackup2016
Running in 'ignore errors' mode due to --ignore-errors; please re-consider if this was not intended
Warning, found incomplete backup sets, probably left from aborted session
Last full backup date: none
Collection Status
-----------------
Connecting with backend: BackendWrapper
Archive directory: /home/hamish/.cache/duplicity/37e66337a462669832db8b2b852f9c6f

Found 0 secondary backup chains.
No backup chains with active signatures found
Also found 0 backup sets not part of any chain,
and 2 incomplete backup sets.
These may be deleted by running duplicity with the "cleanup" command.


% duplicity --no-encryption --ignore-errors list-current-files file:///home/hamish/poonbackup2014
Running in 'ignore errors' mode due to --ignore-errors; please re-consider if this was not intended
Synchronising remote metadata to local cache...
Deleting local /home/hamish/.cache/duplicity/f152114ed2326b0ba48e42e6ec0a23d6/duplicity-full.20140915T183744Z.manifest (not authoritative at backend).
Last full backup date: none
Traceback (innermost last):
  File "/usr/bin/duplicity", line 1555, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1541, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1393, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1476, in do_backup
    list_current(col_stats)
  File "/usr/bin/duplicity", line 702, in list_current
    sig_chain = col_stats.get_signature_chain_at_time(time)
  File "/usr/lib/python2.7/dist-packages/duplicity/collections.py", line 998, in get_signature_chain_at_time
    raise CollectionsError("No signature chains found")
 CollectionsError: No signature chains found

Assuming I cannot extract any more files from the original hard drive, do I have any options to extract files?

I am most interested in extracting photos - if I could extract a file of photos with no names or file metadata that would be a win at this point.

1

collection status finds a chain in file:///home/hamish/poonbackup2014 . so theoretically you should be able to restore the whole backup. signatures are only needed to restore single files/folders or list the contents.

..ede/duply.net

1

I overall followed the guide in this answer but I wanted to expand on some points:

Overall

The difftar.gz files are all just plain tar.gz files, no encryption. And if you unpack them all they put their content into a file structure. The top level has two directories:

  • snapshot/ - this contains plain files, but in my case it had a (very) small subset of the files I cared about - about 100 MB of the 100 GB I wanted to recover.
  • multivol_snapshot/ - this contains a directory for each plain file. In each directory there are a set of files that have just a number as a filename - 1, 2, ... The filesize appears to always be 64 kB (apart from the last one of course). You can manually cat the files together, which would work reasonably for a small number of files. I had a lot though.

Unpack

First I needed to unpack all the tar gz files:

for f in duplicity-full.*.difftar.gz; do echo "$f"; tar xf "$f"; done

Reconstruct

The answer liked above has a shell one liner, but that creates lots of files named content mixed in with the original content. I wanted to recreate the filesystem. So I wrote the below python script, which takes 2 arguments:

  • a directory inside multivol_snapshot/ containing the bits you want to restore
  • a directory to put the restored files inside.

It will recursively go through everything and recreate the files. Provided you have all the chunks required, it will work fine ...

#!/usr/bin/env python3
import argparse
from pathlib import Path
import shutil
import sys


class FileReconstructor():

    def __init__(self, unpacked_dir, restore_dir):
        self.unpacked_path = Path(unpacked_dir).resolve()
        self.restore_path = Path(restore_dir).resolve()

    def reconstruct_files(self):
        for leaf_dir in self.walk_unpacked_leaf_dirs():
            target_path = self.target_path(leaf_dir)
            target_path.parent.mkdir(parents=True, exist_ok=True)
            with target_path.open('wb') as target_file:
                self.copy_file_parts_to(target_file, leaf_dir)

    def copy_file_parts_to(self, target_file, leaf_dir):
        file_parts = sorted(leaf_dir.iterdir(), key=lambda x: int(x.name))
        for file_part in file_parts:
            with file_part.open('rb') as source_file:
                shutil.copyfileobj(source_file, target_file)

    def walk_unpacked_leaf_dirs(self):
        """
        based on the assumption that all leaf files are named as numbers
        """
        seen_dirs = set()
        for path in self.unpacked_path.rglob('*'):
            if path.is_file():
                if path.parent not in seen_dirs:
                    seen_dirs.add(path.parent)
                    yield path.parent

    def target_path(self, leaf_dir_path):
        return self.restore_path / leaf_dir_path.relative_to(self.unpacked_path)


def parse_args(argv):
    parser = argparse.ArgumentParser()
    parser.add_argument(
        'unpacked_dir',
        help='The directory with the unpacked tar files',
    )
    parser.add_argument(
        'restore_dir',
        help='The directory to restore files into',
    )
    return parser.parse_args(argv)


def main(argv):
    args = parse_args(argv)
    reconstuctor = FileReconstructor(args.unpacked_dir, args.restore_dir)
    return reconstuctor.reconstruct_files()


if __name__ == '__main__':
    sys.exit(main(sys.argv[1:]))

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.