2

Wondering if I can get a hand here parsing the output from the command duplicity collection-status MYPATH

My ulitimate goal is to build a "restore" script that will use this command and build out a selectable list of backups to restore from.

Right now, my code is as follows:

    # get the backups information
    _t=`duplicity collection-status $1`

    # split the collection based on new lines
    IFS=$'\n' _arr=($_t)

    # find the length of the array
    _arr_len=${#_arr[@]}      

    # we only want the last set of lines, minus the last 2 lines
    _end=`expr $_arr_len - 2`

    # loop over our resulting set and echo out the line
    _idx=1
    _menu_idx=1
    _ret_arr=()

    # make sure there are actually backups created
    if [[ -z ${_arr[14]} ]]; then
        # it doesnt exist, show a message and exit
        echo 
        echo '--------------------------------------------------------------------';
        echo 'There are no backups for that account/app'
        echo 'Please create the account, and make sure it matches the restore'
        echo 'Account and App names'
        echo '--------------------------------------------------------------------';
        echo 
        exit 1;
    fi;

    echo 
    echo '--------------------------------------------------------------------';
    echo "- Select a restore point: "

    for _l in ${_arr[@]}; do
        if [ $_idx -ge 14 -a $_idx -le $_end ]; then

            IFS=$' ' _temp_arr=($_l)
            _d_string=${_temp_arr[1]}" "${_temp_arr[2]}" "${_temp_arr[3]}" "${_temp_arr[4]}" "${_temp_arr[5]}
            _ret_arr+=( "$_d_string" )
            # Tue Aug 27 10:59:43 2019
            echo $_menu_idx") "$_d_string;

            ((_menu_idx=_menu_idx+1))
        fi;
        ((_idx=_idx+1))
    done
    read n

    # get the value of the selected item
    _t=${_ret_arr[$n-1]};

    # make sure the selection is valid
    if [[ -z $_t ]]; then
        echo "You selected an invalid restore point.  Please try again"
        exit 1;
    fi;

    echo 

    # convert the selected value to a epoch date and return it
    _ret_date=$(date -d "${_t}" +"%s");

And while this appears to do the trick for my purposes, I have found on a couple different servers that I am ending up with some "extra" text before and after so the listing is not always just the backup points. An example output is structured somewhat like this:

root@sp-stage1:~# duplicity collection-status $_dest/$_host/apps/aats/aats/
Synchronizing remote metadata to local cache...
Copying duplicity-full-signatures.20190903T134927Z.sigtar.gz to local cache.
Copying duplicity-full.20190903T134927Z.manifest to local cache.
Copying duplicity-inc.20190903T134927Z.to.20190903T162118Z.manifest to local cache.
Copying duplicity-inc.20190903T162118Z.to.20190904T050005Z.manifest to local cache.
Copying duplicity-inc.20190904T050005Z.to.20190905T050004Z.manifest to local cache.
Copying duplicity-inc.20190905T050004Z.to.20190906T050005Z.manifest to local cache.
Copying duplicity-inc.20190906T050005Z.to.20190907T050005Z.manifest to local cache.
Copying duplicity-inc.20190907T050005Z.to.20190908T050004Z.manifest to local cache.
Copying duplicity-new-signatures.20190903T134927Z.to.20190903T162118Z.sigtar.gz to local cache.
Copying duplicity-new-signatures.20190903T162118Z.to.20190904T050005Z.sigtar.gz to local cache.
Copying duplicity-new-signatures.20190904T050005Z.to.20190905T050004Z.sigtar.gz to local cache.
Copying duplicity-new-signatures.20190905T050004Z.to.20190906T050005Z.sigtar.gz to local cache.
Copying duplicity-new-signatures.20190906T050005Z.to.20190907T050005Z.sigtar.gz to local cache.
Copying duplicity-new-signatures.20190907T050005Z.to.20190908T050004Z.sigtar.gz to local cache.
Last full backup date: Tue Sep  3 09:49:27 2019
Collection Status
-----------------
Connecting with backend: BackendWrapper
Archive dir: /home/ubuntu/.cache/duplicity/6a9fb8e3df936035e1be5ede96c0d7a7

Found 0 secondary backup chains.

Found primary backup chain with matching signature chain:
-------------------------
Chain start time: Tue Sep  3 09:49:27 2019
Chain end time: Sun Sep  8 01:00:04 2019
Number of contained backup sets: 7
Total number of contained volumes: 8
 Type of backup set:                            Time:      Num volumes:
                Full         Tue Sep  3 09:49:27 2019                 2
         Incremental         Tue Sep  3 12:21:18 2019                 1
         Incremental         Wed Sep  4 01:00:05 2019                 1
         Incremental         Thu Sep  5 01:00:04 2019                 1
         Incremental         Fri Sep  6 01:00:05 2019                 1
         Incremental         Sat Sep  7 01:00:05 2019                 1
         Incremental         Sun Sep  8 01:00:04 2019                 1
-------------------------
No orphaned or incomplete backup sets found.

In this specific example, the menu produced is like this:

--------------------------------------------------------------------
- Select a restore point:
1) of contained backup sets: 8
2) number of contained volumes: 9
3) of backup set: Time: Num
4) Tue Sep 3 09:49:27 2019
5) Tue Sep 3 12:21:18 2019
6) Wed Sep 4 01:00:05 2019
7) Thu Sep 5 01:00:04 2019
8) Fri Sep 6 01:00:05 2019
9) Sat Sep 7 01:00:05 2019
10) Sun Sep 8 01:00:04 2019
11) Sun Sep 8 09:33:26 2019

When it really only should be:

--------------------------------------------------------------------
- Select a restore point:
1) Tue Sep 3 09:49:27 2019
2) Tue Sep 3 12:21:18 2019
3) Wed Sep 4 01:00:05 2019
4) Thu Sep 5 01:00:04 2019
5) Fri Sep 6 01:00:05 2019
6) Sat Sep 7 01:00:05 2019
7) Sun Sep 8 01:00:04 2019
8) Sun Sep 8 09:33:26 2019

(yes the double backups on Sept 8th are correct... for now)

So... I must be doing it wrong to show those extra few lines... how can I fix this, and where exactly am I going wrong?

(would be so much easier if I could simply pass a --json flag, but that has not been implemented as of yet...)

1

Where exactly am I going wrong?

if [ $_idx -ge 14 -a $_idx -le $_end ]

The entire approach of parsing output by counting specific numbers of non-blank lines from the beginning and end. It is highly fragile, and obviously cannot cope with a different number of lines being returned before and after the ones you wanted.

Your provided input from duplicity has a lot more than 3 more than your expected 14 non-empty lines before the wanted data. One possibility is this input could have been a mix of stdout and stderr, whereas your code only reads stdout. However, I note there are 14 non-blank lines excluding the Copying lines. As the input from duplicity also does not match the expected or actual output for the backup dates, I suspect there were only three Copying lines in the actual input for your example.

If so, and if the format of the output is otherwise the same in all cases, your code might work if duplicity collection-status is run twice without a new backup in between. But this also assumes there will be no new surprises in the output, including possible changes in any future version of duplicity.

Rather than counting, scan for lines beginning 'Full' or 'Incremental' instead. Extra output is less likely and it's also much simpler.

It's an important habit to always double quote "$variables" except when you are intentionally word breaking (e.g. your IFS lines). If you only do the minimum, it's easy to be caught out by unexpected word breaking later on future input you hadn't originally anticipated.

Also, the bash built-in select (see help select) is an easy way to handle menus.

# get the backups information
_t=`duplicity collection-status "$1"`

# split the collection based on new lines
IFS=$'\n' _arr=($_t)

# loop over our resulting set and select lines
_ret_arr=()
for _l in "${_arr[@]}"; do
    IFS=$' ' _temp_arr=($_l)
    if [ Full = "${_temp_arr[0]}" -o Incremental = "${_temp_arr[0]}" ]; then
        _d_string="${_temp_arr[1]} ${_temp_arr[2]} ${_temp_arr[3]} ${_temp_arr[4]} ${_temp_arr[5]}"
        _ret_arr+=( "$_d_string" )
    fi;
done

echo 
echo '--------------------------------------------------------------------';
# make sure there are actually backups created, i.e. at least one line matched
if [[ -z "${_ret_arr[0]}" ]]; then
    # it doesnt exist, show a message and exit
    echo 'There are no backups for that account/app'
    echo 'Please create the account, and make sure it matches the restore'
    echo 'Account and App names'
    echo '--------------------------------------------------------------------';
    echo 
    exit 1;
fi

echo "- Select a restore point: "
select _t in "${_ret_arr[@]}"; do
    # will keep asking until valid value given
    [ -n "$_t" ] && break
done

echo 

# convert the selected value to a epoch date and return it
_ret_date=$(date -d "${_t}" +"%s")
  • thank you for the answer and the condescending remarks. Highly appreciated ... ( insert sarcastic tone here ) I'd like to know how you think my "entire approach is wrong" and why you think "other" servers actually matter in the context of this question? – Kevin Sep 16 at 16:19
  • 1
    Okay, you're right, I should have worded that a lot better. Also I may have misunderstood your "I have found on a couple different servers that I am ending up with some extra text" as 'only on some servers' not 'testing on various servers'. – Martin Thornton Sep 16 at 22:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.