I want to run a command/shell script say script.sh from 1AM to 2AM per minute. After observing that crontab can't work it out 'per sec' I had to go with 'per min'.

Basically, My server.sh file takes backup of my database on server. This is done by three text files with names of tables in it, hence creating three backup archives. These backup archives are named like 'prefixYYYYmmdd', where prefix = heavy/light/weekly according to the backup type.

Now I want to copy these particular files after they are stored on server. Suppose the backup on server took 15 min. Now I should run the copy script-client.sh on my local computer. As this copy script doesn't know the names of the files just created by server, it should run per sec/min in the interval those files are being created by server and grab that file.

How to use crontab per min/sec within that time interval?

  • 2 questions, why not just download after 2 AM? and why not push the backups from server to client? that way you know when to push – Karthik T Oct 8 '12 at 10:06
  • Downloading backup from server is nothing but copying it, but I don't know what to copy because I don't know the names of those three files and I just can't copy the whole folder from server because it may contain a lot of data. pushing the backup from server to client is a better idea. But what if server is overwhelmed by the backup process. – Hussain Tamboli Oct 8 '12 at 10:17
  • 1
    server can call push after the backup, i would expect it would be more efficient that continuous polling to see if the backup is ready – Karthik T Oct 8 '12 at 10:48
  • p.s i not sure i understand how your script knows what to copy if you dont know the file name and if copying the whole folder isnt the answer. Do you copy the latest created file or something? If this is the case you can use static names and just copy once? – Karthik T Oct 8 '12 at 10:49

First make a cron job that will start by 1 AM

 1 * * * /path_to_script/script.sh

In script.sh

while true
find dir -mtime -1 -exec cp {} directory \;
TEST=`echo $(date +%H)`
if [ $TEST > 2 ]
  exit 0

sleep 1

Modify the find command to your need ( To find the files modified recently use mtime and use rsync if you want to back from a remote machine or just cp/mv in case it from a local machine)

This script will check for the date and if its beyond 2 AM it will exit automatically.

Hope this helps

  • looks good. Will it work for scp? my data is on remote server. I have used ssh-keygen for the password less scp. Used my public key for the authentication. see- linuxquestions.org/questions/linux-general-1/… – Hussain Tamboli Oct 8 '12 at 12:39
  • Tell me if I'm wrong - do ya mean dir: <remote dir>, {}: <for files>(how will it identify the file by name) and directory: <for local dir>. Mind my writing. – Hussain Tamboli Oct 8 '12 at 12:57
  • 1
    Yes you are right. Alter the script to first establish a server connection. Then use find command with mtime option to find the newly created/modified files then do a scp and point a directory(client - local dir). – devav2 Oct 8 '12 at 17:16
  • The cron line is missing the minutes field. I guess it was meant to be " 0 1 * * * /path_to_script/script.sh ". Using {} without quotes will break on file names with spaces. Use "{}" instead. But anyway, doing a find and a cp every second for an hour doesn't look like the right solution to a (poorly explained) backup problem. – mivk Nov 1 '12 at 13:30

There's several approaches you could use.

  1. Get the script to loop 3600 times with a sleep 1 in the middle (= 1 hour) start it with cron at whatever time suits you.
  2. Get the script to intelligently look for the files based on the date, then do the copy at a suitable time eg 4AM
  3. Get the script to monitor when the files are made based on the date and any other relevent information. Do the copy as soon as the files are ready.

Note that "...copy script doesn't know the names of the files just created..." just means that the script doesn't have the code yet. You've said the files with have the current date, just work with that. eg:

FILE=$(date +"prefix%Y%m%d")
echo "The filename for today is "$FILE
  • this is better. Thanks. As @Karthik T said what if we do the copying as soon as the file is created. I think, it will overwhelm the server because we don't know how much time it is going to take before the next backup. – Hussain Tamboli Oct 8 '12 at 10:22
  • can't there be a better way instead iterating 3600 times. – Hussain Tamboli Oct 8 '12 at 10:38
  • 1
    That's up to you, since you're writing the script :-) But yes, options two and three are designed to get away from the brute force method of trying to do whatever it is that you wrote the script for, every second for an hour. I only included it because that's where you were heading initially. – fabricator4 Oct 8 '12 at 10:57
  • right! sorry. I am heading towards a new approach now - I know at what time my server is going to create the files, I will be doing the copying after an hour. This way It will give server enough time to create backup. (my server is taking 10-15min to create one whole backup archive of ~100mb). Off course this is a static way of doing it. – Hussain Tamboli Oct 8 '12 at 11:06

I would use rsync for a job like that. There are even ready made solutions out there. Even back in time, normally used as client-to-local-disk backup AFAIK, would be suitable for your case if you can mount the backup destination via nfs/smb/...

Doing it with an ad hoc solution is fine in the beginning, but needs change inevitable -- and ad hoc solutions are seldom capable of adapting gracefully.

  • 1
    rdiff-backup is something based on rsync that ive used in a similar situation with good effect. It provides historical data as to state of backup at a particular time – Karthik T Oct 8 '12 at 10:46
  • @Tom Regner and @Karthik T, This rsync stuff looks great. Too bad can't upvote you guys. – Hussain Tamboli Oct 8 '12 at 11:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.