running a backup script after dropbox has updated

Question

running: Ubuntu 18.04 (gnome) (also installed xfce4 package, if that matters)

cron version: 3.0pl1-128.1ubuntu1

on: amd64

updates: all packages are up to date and having no issues that I am aware of

kernel: 5.0.0-29-generic #31~18.04.1-Ubuntu

I am trying to run a backup script that I know works because I have tested it. I also know the cronjob will run and the script will execute successfully if I take the while loop out. I wish to keep the while loop because I want the backup to occur after dropbox is up to date. I have placed the echo command in the while loop and then watch -n1 the output of cat on the log file and see that what is logged to the file every 10 seconds is "Dropbox isn't running!" but I know that it is because if I run dropbox status it says "Up to date". Here is the script:

#!/bin/bash

if [[ $(($(date +%A-%B-%d-%Y) != $(cat /home/user/Log/backup.log | tail -n1 | cut -d " " -f7))) \
     -eq 1 ]]
then
    while [[ $(dropbox status) != "Up to date" ]]
    do
        echo $(dropbox status) >> /home/user/Log/backup.log
        sleep 10s
    done
    shopt -s dotglob globstar

    if [[ $(($(date +%d) % 2)) -eq 0 ]]
    then
        rm -rf /backup/daily0/home/* >> /home/user/Log/backup.log 2>&1
        rm -rf /backup/daily0/etc/* >> /home/user/Log/backup.log 2>&1
        cp -r --preserve /home/user/*/backup/daily0/home >> /home/user/Log/backup.log 2>&1
        cp -r --preserve /etc/* /backup/daily0/etc >> /home/user/Log/backup.log 2>&1
        rm -rf /backup/daily0/home/Dropbox/.dropbox >> /home/user/Log/backup.log 2>&1
        rm -rf /backup/daily0/home/Dropbox/.dropbox.cache >> /home/user/Log/backup.log 2>&1
        echo "successful backup of /home/user and /etc: $(date +%A-%B-%d-%Y) $(date +%l:%M%P)" >>\
        /home/user/Log/backup.log
    else 
        rm -rf /backup/daily1/home/* >> /home/user/Log/backup.log 2>&1
        rm -rf /backup/daily1/etc/* >> /home/user/Log/backup.log 2>&1
        cp -r --preserve /home/user/* /backup/daily1/home >> /home/user/Log/backup.log 2>&1
        cp -r --preserve /etc/* /backup/daily1/etc >> /home/user/Log/backup.log 2>&1
        rm -rf /backup/daily1/home/Dropbox/.dropbox >> /home/user/Log/backup.log 2>&1
        rm -rf /backup/daily1/home/Dropbox/.dropbox.cache >> /home/user/Log/backup.log 2>&1
        echo "successful backup of /home/user and /etc: $(date +%A-%B-%d-%Y) $(date +%l:%M%P)" >>\
        /home/user/Log/backup.log
    fi
    exit 0
else
    exit 0
fi

Here is the sudo crontab:

@reboot /usr/local/bin/dailyBackup

I have tried several things:

@reboot sleep 60 && /usr/local/bin/dailyBackup

@reboot /usr/bin/dropbox start && /usr/local/bin/dailyBackup

@reboot /usr/bin/dropbox start && sleep 60 && /usr/local/bin/dailyBackup

I have also tried putting a dropbox start command in the script itself and putting a sleep 60 as the first command in the script, but no matter what I do the output of dropbox status inside the script being run by cron is "Dropbox isn't running!". So cron is running the script at boot or reboot, but the output of the "dropbox status" command is not being updated once dropbox is indeed started. What can I do to remedy this?

edit:

Meant to make clear that the script runs perfectly with the while loop in place when called after boot during normal run of system like:

sudo dailyBackup

Also tried specifying full path to dropbox command in script like:

/usr/bin/dropbox status

which does not work either.

Answer 1

I have figured out how to solve the issue. I read this: https://www.linux.com/news/introduction-services-runlevels-and-rcd-scripts/ and then I copied my script into /etc/init.d and made a symlink to it from /etc/rc5.d as follows:

sudo ln -s ../init.d/dailyBackup ./S01dailyBackup

Now what was just logged to /home/user/Log/backup.log as I watched it after boot was:

Dropbox isn't running!
Checking for changes...
successful backup of /home/user and /etc: Sunday-September-29-2019 at 10:06am

This feels like a more solid solution than running a cronjob at boot anyway. Here is the final version of the script:

#!/bin/bash

if [[ $(($(date +%A-%B-%d-%Y) != $(cat /home/user/Log/backup.log | tail -n1 | cut -d " " -f7))) \
     -eq 1 ]]
    then
    while [[ $(sudo -u user dropbox status) != "Up to date" ]]
    do
        echo $(sudo -u user dropbox status) >> /home/user/Log/backup.log
        sleep 10s
    done
    shopt -s dotglob globstar

    if [[ $(($(date +%d) % 2)) -eq 0 ]]
        then
        if [[ -e /backup/daily0/user ]]
            then
            rm -rf /backup/daily0/user/ >> /home/user/Log/backup.log 2>&1
        fi
        if [[ -e /backup/daily0/etc ]]
            then
            rm -rf /backup/daily0/etc/ >> /home/user/Log/backup.log 2>&1
        fi
        cp -r --preserve /home/user/ /backup/daily0/ >> /home/user/Log/backup.log 2>&1
        cp -r --preserve /etc/ /backup/daily0/ >> /home/user/Log/backup.log 2>&1
        rm -rf /backup/daily0/user/Dropbox/.dropbox >> /home/user/Log/backup.log 2>&1
        rm -rf /backup/daily0/user/Dropbox/.dropbox.cache >> /home/user/Log/backup.log 2>&1
        echo "successful backup of /home/user and /etc: $(date +%A-%B-%d-%Y) $(date +%l:%M%P)" >>\
        /home/user/Log/backup.log
    else 
        if [[ -e /backup/daily1/user ]]
            then
            rm -rf /backup/daily1/user/ >> /home/user/Log/backup.log 2>&1
        fi
        if [[ -e /backup/daily1/etc ]]
            then
            rm -rf /backup/daily1/etc/ >> /home/user/Log/backup.log 2>&1
        fi
        cp -r --preserve /home/user/ /backup/daily1/ >> /home/user/Log/backup.log 2>&1
        cp -r --preserve /etc/ /backup/daily1/ >> /home/user/Log/backup.log 2>&1
        rm -rf /backup/daily1/user/Dropbox/.dropbox >> /home/user/Log/backup.log 2>&1
        rm -rf /backup/daily1/user/Dropbox/.dropbox.cache >> /home/user/Log/backup.log 2>&1
        echo "successful backup of /home/user and /etc: $(date +%A-%B-%d-%Y) $(date +%l:%M%P)" >>\
        /home/user/Log/backup.log
    fi
    exit 0
else
    exit 0
fi

I tried using both 7Z to compress the directories into an archive, which did save around 10GB per backup and since I am keeping 2 days worth at any given time that is 20GB of saved disk space. The issue is that the script takes an hour to run so I then tried it with tar -cjf, which saved around 8GB per backup, but still it took almost an hour for the script to complete its task. So I am sacrificing the 20GB of disk space for efficiency sake, after all Big-O is usually more important these days unless dealing with a very small capacity on embedded systems. I enjoyed solving this and I hope it helps someone trying to do a similar thing sometime:) For me it is nice to have the last 2 days of all those files and directories backed up in case I want to retrieve an older version before I made some change to a school project or a config file in /etc or whatever. Maybe I will add some other directories to the backups in the future as learn more. At my school, all of us computer science students have an account on a Debian based school server, which is really cool. We all have access to 21 days worth of backukps of our home directories in the directories /backup/daily.0/userName through /backup/daily.20/userName, this is where I got the idea after saving my .bash_history from 21 days ago and merging it with my history from one day ago. I wrote a script which keeps my .bash_history and my .tmux_history merged by a cronjob every 5 minutes. while debugging the script I had demolished both file's contents a couple of times and was happy to be able to save my history as I keep an unlimited history file length which I have a script to truncate by removing duplicate entries once a year on a cronjob.

Answer 2

I chose not to edit the previous answer, as suggested by the prompt when I clicked to add this one so that the comments on that answer are not invalidated and because I feel the previous answer and comments are valuable in the learning experience they reflect.

---

I got excited an posted the previous answer before even powering down my machines to see that there was some issue with something still running. I have not figured out exactly what that issue was, I believe it was due to my script not being LSB compliant and being in /etc/init.d.

I have actually solved the problem now and the solution has been working great for a couple of days, I have tested it multiple times in that period.

What I did: I found this post while working on getting this same script to run at boot on my Arch installation, which is on a separate drive in this desktop: https://unix.stackexchange.com/questions/138281/arch-linux-run-script-a-minute-after-boot and then I revisited this page that I have not looked in a while: https://wiki.archlinux.org/index.php/Systemd the following solution worked on my Arch machine and, realizing that Ubuntu also runs Systemd, I decided to try it on Ubuntu as well ...success:)

Here is the now working solution:

#!/bin/bash

if [[ $(($(date +%A-%B-%d-%Y) != $(cat /home/user/Log/backup.log | tail -n1 | cut -d " " -f7))) \
     -eq 1 ]]
    then
    while [[ $(sudo -u user dropbox status) != "Up to date" ]]
    do
        sleep 10s
        [[ $(uptime | cut -d " " -f4) -ge 30 ]] && echo "Dropbox took to long on $(date)" >> \
        /home/user/Log/backup.log && exit 0
    done
    shopt -s dotglob globstar

    # Change to BACKUPNUM=$(($(date +%d) % 10)) when get separate drive to                  
    # backup to some day

    BACKNUM=$(($(date +%d) % 2))
    rsync -aAX --delete /home/user/ /backup/daily${BACKNUM}/user/ >> \
    /home/user/Log/backup.log 2>&1
    [[ $? -eq 0 ]] && rsync -aAX --delete /etc/ /backup/daily${BACKNUM}/etc/ >> \
    /home/user/Log/backup.log 2>&1
    [[ $? -eq 0 ]] && rm -rf /backup/daily${BACKNUM}/user/.dropbox >> \
    /home/user/Log/backup.log 2>&1
    [[ $? -eq 0 ]] && rm -rf /backup/daily${BACKNUM}/user/Dropbox/.dropbox >> \
    /home/user/Log/backup.log 2>&1
    [[ $? -eq 0 ]] && rm -rf /backup/daily${BACKNUM}/user/Dropbox/.dropbox.cache >> \
    /home/user/Log/backup.log 2>&1
    [[ $? -eq 0 ]] && \
    echo "successful backup of /home/user and /etc: $(date +%A-%B-%d-%Y) $(date +%l:%M%P)" >> \
    /home/user/Log/backup.log
    exit 0
else
    exit 0
fi

So I added the following two files /etc/systemd/system/dailyBackup.service:

[Unit]
Description=dailyBackup

[Service]
Type=simple
ExecStart=/usr/local/bin/dailyBackup

and /etc/systemd/system/dailyBackup.timer:

[Unit]
Desciption=Runs dailyBackup one minute after boot

[Timer]
#how long to wait before executing
OnBootSec=1min
Unit=dailyBackup.service

[Install]
WantedBy=multi-user.target

I then ran:

sudo systemctl enable dailyBackup.timer

and shutdown the system. Upon powering back up, the script executed successfully and there are no shutdown or reboot issues. This solution, as earlier stated, has been working for a couple of days now. As can be seen in the script, I am using rsync now which was suggested by bacOn in their comment to my last answer. I have tried rsync a few times in the past and have never really fully figured out why there are still differences when I run diff -r, but I think I have realized that these are due to system .files that have changed slightly since the run of rsync and also because I seem to have some broken symlinks in the ~/snap/gnome-system-monitor/current/.local/share/icons/hicolor on my desktop and there is not even a snap directory on my laptop Ubuntu. This must be due to something I have previously installed and removed. That is for another question though as is the issue that the --delete rsync flag seems to cancel the --exclude flag, even when I add the --delete-excluded flag. Which is why my script still has the rm statements after the rsync. I hope this is helpful to someone, I have learned from this and to me that is invaluable. Now I need to do some actual school work instead of playing with and learning about Linux, which is one of my favorite things to do:)

update:

So I figured out that the patterns in the rsync flag --exclude={} have to be relative paths, relative to the source directory being synced from and that is why the excludes were being ignored entirely. I was putting absolute paths in. so the rm -rf statements are no longer needed. I have also added some conditionals to the while loop now and decreased the sleep to one second( I know this is close to polling the command ) so that if dropbox is taking to long to update because I have added some huge video files or something the script will exit 0 if I shutdown or reboot while the script is still running that while loop. So until I figure out a better way to stop the script in the case of a shutdown or reboot and a better way to have the script wait until dropbox is up to date, perhaps something in dailyBackup.timer to not even run it until dropbox is up to date, here is the current best working version of the script:

#!/bin/bash

if [[ $((10#$(date +%d) != 10#$(cat /home/user/Log/backup.log | tail -n1 | cut -d " " -f7 | \
      cut -d "-" -f3))) -eq 1 ]]
    then
    while [[ $(sudo -u user dropbox status) != "Up to date" ]]
    do
        [[ $(runlevel | cut -d " " -f2) -eq 0 ]] && exit 0
        [[ $(runlevel | cut -d " " -f2) -eq 6 ]] && exit 0
        [[ $(uptime | cut -d " " -f4) -ge 30 ]] && echo "Dropbox took to long on $(date)" >> \
        /home/user/Log/backup.err && exit 0
        sleep 1s
    done
    shopt -s dotglob globstar

    # Change to BACKUPNUM=$(($(date +%d) % 10)) when get separate drive to backup to someday

    BACKNUM=$((10#$(date +%d) % 2))
    rsync -aAX --delete-excluded \
    --exclude={".dropbox","Dropbox/.dropbox",".dropbox-dist","Dropbox/.dropbox.cache"} \
    /home/user/ /backup/daily${BACKNUM}/user/ >> /home/user/Log/backup.err 2>&1
    [[ $? -eq 0 ]] && rsync -aAX --delete /etc/ /backup/daily${BACKNUM}/etc/ >> \
    /home/user/Log/backup.err 2>&1
    [[ $? -eq 0 ]] && \
    echo "successful backup of /home/user and /etc: $(date +%A-%B-%d-%Y) $(date +%l:%M%P)" >> \
    /home/user/Log/backup.log
    exit 0
else
    exit 0
fi

I have not stated it, however to use this I created a /backup directory and then a /backup/daily0 and /backup/daily1 and "sudo chown me:myGroup" both of them and then created a user and an etc directory inside both daily0 and daily1.

Also on my laptop I changed dailyBackup.timer to start the script 3 minutes after boot since it takes longer for wifi to get going than a wired connection and on my desktop I changed it to 2 minutes just to help it not loop as long in the case of a long dropbox update time.

Also when the date hit 08 I found that since the numbers in the first conditional if are being interpreted as octal I had to prepend 10# so that it uses base 10 numbers everywhere which prompted me to just add an additional cut -d to use only the actual day of the month instead of comparing the entire date string. The issue on this website is that the pound signs are interpreted in the code block as the start of a comment so it makes it look like the last of those lines are just comments, they are not. I have also found that it is better to send errors to one file and successful logs to another.

the first line of code after the chebang and the line where the variable BACKNUM is instantiated are the lines I am talking about.