0

I have a bash file which runs a process on a series of newly downloaded files, every day, the total run time for all the processing is several hours.

I want to run a particular bash command 24 hours after a process has been run on a particular file.

I had tried to use at for this, however I don't think you can use at to run a bash command i.e.:

at now +24 hours gsutil rm gs://$google_cloud_storage_location/$processed_file

And I also can't get it to run a bash script with arguments.

at now +24 hours -f example_script.bash $google_cloud_storage_location/$processed_file

What is the correct way to go about scheduling these tasks? Do I just need a completely different tool or am I missing a way with at?

Example bash script:

for file in list_of_files; do
    #do some processing
    ....
    #then delete the file
    at now +24 hours gsutil rm gs://file
done
  • How about at now +24 hours /usr/local/bin/doit, and begin /usr/local/bin/doit with #!/bin/bash, followed by what you want? – waltinator Dec 20 '16 at 0:52
  • @waltinator So I have a loop in by bash file which is running this code and if I don't provide it with the variable then, I'm not sure how I get it to delete the correct file. How I do provide the variable in that example? – Dominic Woodman Dec 20 '16 at 0:55
  • 2
    I think you need to pass the command to at via stdin - either something like echo "command <args>" | at now +24 hours, or by using a heredoc at now +24 hours << EOD; command <args>; EOD – steeldriver Dec 20 '16 at 1:02
  • @steeldriver aaah spot on - pushing the commands through echo worked like charm. if you want post a full answer so I can mark as correct or I can write up myself – Dominic Woodman Dec 20 '16 at 1:06
2

If you want your variables to be substituted at the time of setting the job up, you can accomplish running a simple bash command with at by simply piping the command to at with echo. For your first example, the syntax would be

echo "gsutil rm gs://$google_cloud_storage_location/$processed_file" | at now +24 hours

For the second example, the syntax could be

echo "/path/to/example_script.bash $google_cloud_storage_location/$processed_file" | at now +24 hours

If the variable substitution needs to be performed at the time of the job, it is a bit trickier. You will need to create a script that runs when the job is to be done, substituting at the proper time. This can be done using a command exclusively (e.g. from a script) by doing

echo -e '#!/bin/bash' > /path/to/script

Then

echo '<your command here>' >> /path/to/script

Then the syntax for the at command is

at now +24 hours -f /path/to/script

Of course, the +24 hours part should be adjusted to the proper duration.

1

Since at reads commands from stdin or from a file:

for file in list_of_files; do
    #do some processing
    ....
    #then delete the file
    at now +24 hours <<EOF
gsutil rm gs://"$file"
EOF
done

Using heredocs to provide the commands would work, since variables are expanded in heredocs unless the heredoc marker (EOF, here) is quoted.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.