13

I found this bash script on GitHub that I want to use for my own work. My question is the following: What do the lines 2 to 5 mean? Are they just comments or do they serve any purpose?

#!/bin/bash
#$ -l h_rt=72:00:00
#$ -V
#$ -N index_calc
#$ -j y

source ~/modules.sh

cd $1

l_file=`find . -name 'L*stack' -type f`
for l in $l_file; do
    echo "Running on file:"
    echo $l
      extract=${l:0:45}
    name=${extract}_index.tif
    echo "Name of index stack:"
    echo $name
    echo "Executing code..."
    ~/Documents/misc/spectral/transforms.py \
    -v $l $name evi ndvi nbr ndmi 

done

echo "Done!"
19

The lines beginning with #$ are options for qsub, a command used to submit a job to the SGE cluster, a scheduling system explained in this readme:

Using the BIMSB (soon to be called MAX) cluster environment is similar to using unix/linux environments for your job submission (e.g running your scripts or other software). The difference is that you need to specify needed resources beforehand. The cluster is controlled by a SGE (Sun Grid Engine Software) that organizes the queues and resources. This sort of scheduling system is necessary when limited computational resources are shared by many. And, it would be useful if you are running alignments for multiple samples and want to distribute those tasks (jobs) across multiple machines or CPUs, or when running statistical simulations that needs to run on multiple CPUs for a long time. For these cases and many more alike, you just need to submit your job script (which is a shell script) and Sun Grid Engine will take care of the rest (as long as there is no error in your script).

SGE will do the "job scheduling". That means you can submit all your jobs and SGE will queue them and run them when resources you requested becomes available. SGE will also achieve "load balancing" where the jobs will be distributed so that specific nodes do not get overloaded. In addition, SGE will allow you to do "job monitoring and accounting" which will be useful when you want to check if your job is running ,and if it failed it will help you understand what went wrong.

The syntax of the qsub command is explained on its manpage, your script uses the following options:

  • -l h_rt=<hh:mm:ss> – specify the maximum run time (hours, minutes and seconds)
  • -V – pass all environment variables to the job
  • -N <jobname> – specify the name of the job. This you will see when you use qstat, to check status of your jobs.
  • -j y[es]|n[o] – specifies whether or not the standard error stream of the job is merged into the standard output stream

As How to submit a job using qsub explains one can set the qsub options directly in the script in lines that begin with #$. This an alternative to passing them with the qsub command on the command-line.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.