2

I am writing a small script to upload the files to remote ssh server and on the remote ssh server I will have almost 2 TB of Space available.

I need to check within the shell script if the Space is around 1.5 TB full than sleep the script and **start the script again** once its goes back it goes back to 1TB

here is my part of the code which needs to run the logic for checking the directory size and then put it into the if else statements

SIZE="$(ssh archiveupload@REMOTESERVER-IP "du -s")"
LIMIT="1.5TB"
if [ $SIZE -gt $LIMIT ];
then
  sleep 4h
elif
 continue ;
fi

UPDATE SCRIPT:

if [ ! -f /home/user/Tupload/upload-lock ]; then
    touch /home/user/Tupload/upload-lock
    ###Define the File in unix style#############
    filename="/home/user/Tupload/20186.txt"
    ####starting of the while loop.###########
    LIMIT='1500000000'
    while read line;
    do
        name="${line%$'\r'}"
        SIZE="$(ssh -n USER@REMOTEIP"df /var/www/Upload/" | awk 'NR > 1 {print $4; exit}')"

        if [[ $SIZE -gt $LIMIT ]]
        then
            YEAR=$(echo $name | cut -c 1-4)   #####define the Year from the read line ####
            MONTH=$(echo $name | cut -c 5-6)  #####Define the Month from the read line####
            DAY=$(echo $name | cut -c 7-8)  #####Define the DAY from the read line####
            YM=${YEAR}-${MONTH} ####define the Year and Month######

            april="2017-04"
            may="2017-05"
            june="2017-06"
            march="2017-03"
            ######if else statement ######
            if [ "$april" = "$YM"  ]; then
                cd /mnt/smxfxml/$YEAR/April/
            elif [ "$may" = "$YM"  ]; then
                cd /mnt/smxfxml/$YEAR/May/
            elif [ "$june" = "$YM"  ]; then
                cd /mnt/smxfxml/$YEAR/June/
            elif [ "$march" = "$YM"  ]; then
                cd /mnt/smxfxml/$YEAR/March/
            else
                cd /mnt/smxfxml/$YEAR/$YM/
            fi

            ssh USER@REMOTEIP"mkdir $YEAR/$MONTH/$DAY/$name/" < /dev/null
            scp $name.mxf USER@REMOTEIP:$YEAR/$MONTH/$DAY/$name/
            cd /mnt/smxfxml/XML/$YEAR/$YM/
            scp $name.xml USER@REMOTEIP:$YEAR/$MONTH/$DAY/$name/
        else 
            echo 'Not enough space.'; sleep 4h
        fi
    done <"$filename"
    rm -rf /home/user/Tupload/upload-lock
    exit 0
else
    sleep 0 
fi
exit 0
2
  • 1
    Just a note ssh -n ... is equivalent of ssh ... < /dev/null, reference: askubuntu.com/a/979457/566421
    – pa4080
    Aug 2, 2018 at 5:44
  • Thanks. I figured as well. But the old habits. ;
    – kunal
    Aug 2, 2018 at 5:46

2 Answers 2

2

Thanks to pa4080 for the "free space" one liner.


Now that you shared your script:

It's going to be a little bit more difficult because we cannot use continue inside the while read line loop without skipping lines.

You want to pause the script when there is no more than 0.5TB free, and resume when there is at least 1TB free.

To keep the script short and human readable, define a function that tells you if there is enough space on the server and takes the minimum size as parameter:

enough_space () {
    local limit=$1
    free_space="$(ssh -n user@host "df /" | awk 'NR > 1 {print $4; exit}')"
    [ $free_space -ge $limit ]
}

...and either sleep while not enough space :

while ! enough_space $limit; do
    sleep 4h
done

...or sleep until enough space :

until enough_space $limit; do
    sleep 4h
done

Incorporating this in your script

Put this at the beginning of your script:

pause_limit='500000000'    # 0.5TB free space
resume_limit='1000000000'  # 1.0TB free space

enough_space () {
    local limit=$1
    free_space="$(ssh -n user@host "df /" | awk 'NR > 1 {print $4; exit}')"
    [ $free_space -ge $limit ]
}

We'll introduce an "infinite" while loop inside the existing while loop. This enables us to retry the same file again when there wasn't enough free space the first time. Also, assigning the variables is not dependent on the server's free space, so it should be kept outside of the if statement (and the inner while loop).

####starting of the while loop.###########
while read line; do 
#### this can all be done before checking the server's free space
#    name="${line%$'\r'}"
#    YEAR=$(echo $name | cut -c 1-4) 
#      .
#      .
    ######if else statements ######
#    if [ "$april" = "$YM"  ]; then
#      .
#      .
#    fi

    while true; do
        if enough_space $pause_limit;
        then
            ssh USER@REMOTEIP"mkdir $YEAR/$MONTH/$DAY/$name/" < /dev/null
            scp $name.mxf USER@REMOTEIP:$YEAR/$MONTH/$DAY/$name/
            cd /mnt/smxfxml/XML/$YEAR/$YM/
            scp $name.xml USER@REMOTEIP:$YEAR/$MONTH/$DAY/$name/
            break
        else
            until enough_space $resume_limit; do
                sleep 4h
            done
        fi
    done
done <"$filename"

The "infinite" while loop will either run once (if there is enough free space on the server) or twice (if there is enough free space again).


Additional notes

If you want to run your copy routine over and over again indefinitely, use an infinite while loop (while true; do...). Don't do recursive calls (calling the script within the script or the function within the function), because that will increase the call stack size and your script will crash sooner or later.

This is, of course, only one of many ways to achieve the same thing. One downside of this approach is that whenever the free space falls below 0.5TB or exceeds 1.0TB, there will be one unnecessary call to enough_space.

If I misinterpreted your question, please tell me so I can adapt my answer.

11
  • thank you @danzel am trying to Modify my script according to your answer, Please check my complete script in the question again,
    – kunal
    Aug 1, 2018 at 11:11
  • there is some issue with the script and I can't make it run and copy the files
    – kunal
    Aug 1, 2018 at 13:42
  • @kunal ... what is the issue?
    – danzel
    Aug 1, 2018 at 13:50
  • pastebin.ubuntu.com/p/QDKXCGcwQP
    – kunal
    Aug 1, 2018 at 14:10
  • Please check the Paste Bin for the Output of bash Script
    – kunal
    Aug 1, 2018 at 14:11
2

First I wold suggest you to use the command df to get the free space. Also you should provide a system path (for example /) or device name (for example /dev/sda1) to make the things simple. The output of the command df / executed on my VPS looks like:

$ ssh -n user@host "df /"
Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/vda1       25671932 8335064  16009768  35% /

We need only the available space so we can filter the output by the help of awk (for example):

$ ssh -n user@host "df /" | awk 'NR > 1 {print $4; exit}'
16009768

Second use the same units as the output of the above command (1K-blocks) for the value of the variable $LIMIT: 1.5 TB = 1500000000 KB.

Third put the main logic into a function that will call itself recursively again and again after the sleep command. In addition the places of sleep and continue should be inverted, I think :)

According to the above the script could be something as this:

#!/bin/bash -e

LIMIT='1500000000'

# define the function
main() {
    SIZE="$(ssh -n user@host "df /" | awk 'NR > 1 {print $4; exit}')"

    if [[ $SIZE -gt $LIMIT ]]
    then 
        echo 'continue'
    else
        echo 'Not enough space.'; sleep 4h
        # call the function
        main
    fi
}

# initial call of the function
main

According to the @danzel's comment instead of recursion we can use infinite while loop:

#!/bin/bash -e

LIMIT='1500000000'

while true
do
    SIZE="$(ssh -n user@host "df /" | awk 'NR > 1 {print $4; exit}')"

    if [[ $SIZE -gt $LIMIT ]]
    then
        # remove the exit command if you do not want to interupt the loop
        echo 'continue'; exit
    else
        echo 'Not enough space.'; sleep 4h
    fi
done
3
  • Good answer, except for the third point. I don't know how bash handles this, but in most programming languages I know this will eventually result in a stack overflow. A simple (infinite) while loop would be better than recursion in this case.
    – danzel
    Aug 1, 2018 at 7:23
  • ` sh -x on_news.sh + [ ! -f /home/user/upload-lock ] + touch /home/user/upload-lock + filename=/home/user/20186.txt + LIMIT=1500000000 + read line + name=20180320-A + ssh -n user@host df /var/www/Upload/ + awk NR > 1 {print $4; exit} + SIZE=2310513500 + [[ 2310513500 -gt 1500000000 ]] on_news.sh: 14: on_news.sh: [[: not found + echo Not enough space. Not enough space. + sleep 4h`
    – kunal
    Aug 1, 2018 at 11:02
  • @p4080 , Please check the script above I have modified it as per your answer, but it doesn't work,
    – kunal
    Aug 1, 2018 at 11:31

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .