18

I have found a few similar cron job scripts but nothing exactly how I need them and I do not know enough about scripting for Linux to try modify the code when it comes to this sort of job which could turn disastrous.

Essentially I have ip cameras that record to /home/ben/ftp/surveillance/ but I need to ensure there is always enough space on the disk to do so.

Would someone please be able to guide me how I can setup a cron job to:

Check if /dev/sbd/ has reached 90% capacity. If so then delete the oldest file in(and files in sub folders) /home/ben/ftp/surveillance/ And repeat this until /dev/sbd/ capacity is below 80% Repeat every 10 minutes.

2
  • What if /dev/sdb (note : sdb != sbd - watch for typos in your scripts!) ever becomes >80% full regardless of the contents of /home/ben/ftp/surveillance/? This will always delete all of your recordings, leaving you with nothing. Better to have CCTV cameras recording to a dedicated volume that is not shared with (ie) your operating system or any other users. Ideally the cameras should manage this themselves, being aware of which files are theirs and overwriting their own oldest files when the dedicated drive space becomes full.
    – J...
    Jan 13 at 20:09
  • Good point J... My ftp directory is actually an external drive mounted in my home dir and is now dedicated to just cam recordings. I haven't bothered to change the folder structure it has previous to being a dedicated disk. The cameras also upload a .jpg still of every recording so I will setup another cron to delete all .jpg periodically.
    – Beno
    Jan 14 at 0:52

3 Answers 3

33

Writing these kinds of scripts for people always makes me nervous because, in the event anything goes wrong, one of three things will happen:

  1. I'll kick myself for what's probably a n00b-level typo
  2. Death threats will come my way because someone blindly copy/pasted without:
    • making an effort to understand the script
    • testing the script
    • having a reasonable backup in place
  3. All of the above

So, to reduce the risk of all three, here is a starter kit for you:

#!/bin/sh
DIR=/home/ben/ftp/surveillance
ACT=90
df -k $DIR | grep -vE '^Filesystem' | awk '{ print $5 " " $1 }' | while read output;
do
  echo $output
  usep=$(echo $output | awk '{ print $1}' | cut -d'%' -f1  )
  partition=$(echo $output | awk '{ print $2 }' )
  if [ $usep -ge $ACT ]; then
    echo "Running out of space \"$partition ($usep%)\" on $(hostname) as on $(date)"
    oldfile=$(ls -dltr $DIR/*.gz|awk '{ print $9 }' | head -1)
    echo "Let's Delete \"$oldfile\" ..."
  fi
done

THINGS TO NOTE:

  1. This script deletes nothing

  2. DIR is the directory to work with

  3. ACT is the minimum percentage required to act

  4. Only one file – the oldest – is selected for "deletion"

  5. You will want to replace *.gz with the actual file type of your surveillance videos.
    DO NOT USE *.* OR * BY ITSELF!

  6. If the partition containing DIR is at a capacity greater than ACT, you will see a message like this:

    97% /dev/sda2
    Running out of space "/dev/sda2 (97%)" on ubuntu-vm as on Wed Jan 12 07:52:20 UTC 2022
    Let's Delete "/home/ben/ftp/surveillance/1999-12-31-video.gz" ...
    

    Again, this script will not delete anything.

  7. If you are satisfied with the output, then you can continue to modify the script to delete/move/archive as you see fit

Test often. Test well. And remember: When putting rm in a script, there is no undo.

9
  • 1
    It's silly to pipe grep to awk, you can just add !/^Filesystem/ to the beginning of the awk command. Or you can tell df to only produce the output you want, and use sed to strip the header and percent sign: usep=$(df --output=pcent $DIR | sed '1d;s/%//') Jan 12 at 18:27
  • 1
    Similar, but using the firstaction feature of logrotate: serverfault.com/questions/372809/… Jan 12 at 23:01
  • 5
    @NickMatteo If the corresponding code works fine and is readable, it's silly to argue that it's silly because it could be written differently with a 45-year-old-and-not-very-readable-language. UNIX pipes are awesome and readable, who cares if the command could be written with one or two fewer pipes? Jan 13 at 15:20
  • 1
    @EricDuminil: The somewhat common use of grep 'PATTERN' | awk '{do stuff to matching lines}' is objectively silly, since awk's whole purpose is to do stuff to matching lines, and you could have just written awk '/PATTERN/ {do stuff}'. Jan 13 at 19:30
  • 4
    @NickMatteo: You're free to use whatever tools you're familiar with. matigo seems to be happy with grep ... | awk ..., you'd be happy with awk '/PATTERN/ {do stuff}, and I'd be happy with a short Ruby or Python script. As long as the scripts are working and readable, nobody's wrong, and no command is silly, and surely not objectively silly. Jan 13 at 20:01
12

I would use Python for such task. It might lead to more code than a pure bash solution, but:

  • it's (IMO) easier to test, just use pytest or unitest module
  • it's readable for non Linux people (well except the get_device function which is Linux specific...)
  • it's easier to get started (again IMO)
  • What if you want to send some emails ? To trigger new actions ? Scripts can be enriched easily with a programming language like Python.

Since Python 3.3, shutil module comes with a function named disk_usage. It can be used to get the disk usage based on a given directory.

The minor problem is that I don't known how to easily get the name of the disk, I.E, /dev/sdb, even though it's possible to get its disk usage (using any directory mounted on /dev/sdb, in my case $HOME for example). I wrote a function called get_device for this purpose.

#!/usr/bin/env python3
import argparse
from os.path import getmtime
from shutil import disk_usage, rmtree
from sys import exit
from pathlib import Path
from typing import Iterator, Tuple


def get_device(path: Path) -> str:
    """Find the mount for a given directory. This is needed only for logging purpose."""
    # Read /etc/mtab to learn about mount points
    mtab_entries = Path("/etc/mtab").read_text().splitlines()
    # Create a dict of mount points and devices
    mount_points = dict([list(reversed(line.split(" ")[:2])) for line in mtab_entries])
    # Find the mount point of given path
    while path.resolve(True).as_posix() not in mount_points:
        path = path.parent
    # Return device associated with mount point
    return mount_points[path.as_posix()]


def get_directory_and_device(path: str) -> Tuple[str, Path]:
    """Exit the process if directory does not exist."""
    fs_path = Path(path)
    # Path must exist
    if not fs_path.exists():
        print(f"ERROR: No such directory: {path}")
        exit(1)
    # And path must be a valid directory
    if not fs_path.is_dir():
        print(f"Path must be a directory and not a file: {path}")
        exit(1)
    # Get the device
    device = get_device(fs_path)

    return device, fs_path


def get_disk_usage(path: Path) -> float:
    # shutil.disk_usage support Path like objects so no need to cast to string
    usage = disk_usage(path)
    # Get disk usage in percentage
    return usage.used / usage.total * 100


def remove_file_or_directory(path: Path) -> None:
    """Remove given path, which can be a directory or a file."""
    # Remove files
    if path.is_file():
        path.unlink()
    # Recursively delete directory trees
    if path.is_dir():
        rmtree(path)


def find_oldest_files(
    path: Path, pattern: str = "*", threshold: int = 80
) -> Iterator[Path]:
    """Iterate on the files or directories present in a directory which match given pattern."""
    # List the files in the directory received as argument and sort them by age
    files = sorted(path.glob(pattern), key=getmtime)
    # Yield file paths until usage is lower than threshold
    for file in files:
        usage = get_disk_usage(path)
        if usage < threshold:
            break
        yield file


def check_and_clean(
    path: str,
    threshold: int = 80,
    remove: bool = False,
) -> None:
    """Main function"""
    device, fspath = get_directory_and_device(path)
    # shutil.disk_usage support Path like objects so no need to cast to string
    usage = disk_usage(path)
    # Take action if needed
    if usage > threshold:
        print(
            f"Disk usage is greather than threshold: {usage:.2f}% > {threshold}% ({device})"
        )
    # Iterate over files to remove
    for file in find_oldest_files(fspath, "*", threshold):
        print(f"Removing file {file}")
        if remove:
            remove_file_or_directory(file)


def main() -> None:

    parser = argparse.ArgumentParser(
        description="Purge old files when disk usage is above limit."
    )

    parser.add_argument(
        "path", help="Directory path where files should be purged", type=str
    )
    parser.add_argument(
        "--threshold",
        "-t",
        metavar="T",
        help="Usage threshold in percentage",
        type=int,
        default=80,
    )
    parser.add_argument(
        "--remove",
        "--rm",
        help="Files are not removed unless --removed or --rm option is specified",
        action="store_true",
        default=False,
    )

    args = parser.parse_args()

    check_and_clean(
        args.path,
        threshold=args.threshold,
        remove=args.remove,
    )


if __name__ == "__main__":
    main()

If you need to orchestrate many tasks using CRON, it might be worth putting together some Python code as a library, and reuse this code across many tasks.

EDIT: I finally added the CLI part in the script, I think I'll use it myself 😅

4
  • 1
    fwiw you can send emails from command line.
    – qwr
    Jan 13 at 10:37
  • I'm not saying that it cannot be done in CLI, i'm saying that OP is not familiar with bash, and tant he might find it easier to do it in Python
    – gcharbon
    Jan 13 at 12:00
  • Thanks for the comprehensive post! I do like the more readable a approach but unfortautely I am not so familiar with phython either. But you have made me realise I could do this with php which I am a lot more comfortable with.
    – Beno
    Jan 14 at 0:44
  • +1 for something easier to test. I'd want unit tests for a script like this.
    – Clumsy cat
    Jan 14 at 9:19
1

Check if /dev/sbd/ has reached 90% capacity. If so then delete the oldest file in(and files in sub folders) /home/ben/ftp/surveillance/ And repeat this until /dev/sbd/ capacity is below 80% Repeat every 10 minutes.

The script below will do exactly that (provided that you add it to your crontab to run in 10 minute intervals). Be extra sure this is what you really want to do, since this could easily erase all files in /home/ben/ftp/surveillance/ if your disk is filling up somewhere outside this directory.

#!/bin/sh
directory='/home/ben/ftp/surveillance'
max_usage=90
goal_usage=80
[ -d "$directory" ] || exit 1
[ "$max_usage" -gt "$goal_usage" ] || exit 1
[ "$( df --output=pcent $directory | \
    grep -Ewo '[0-9]+' )" -ge "$max_usage" ] || exit 0
dev_used="$( df -B 1K --output=used $directory | \
    grep -Ewo '[0-9]+' )"
goal_usage="$( printf "%.0f" \
    $( echo ".01 * $goal_usage * \
    $( df -B 1K --output=size $directory | \
        grep -Ewo '[0-9]+' )" | bc ) )"
echo "$( find $directory -type f -printf '%Ts,%k,\047%p\047\n' )" | \
    sort -k1 | \
        awk -F, -v goal="$(($dev_used-$goal_usage))" '\
            (sum+$2)>goal{printf "%s ",$3; exit} \
            (sum+$2)<=goal{printf "%s ",$3}; {sum+=$2}' | \
                xargs rm

How this script works:

The first 3 lines after the shebang are the variables per your parameters:

  • directory is the full path to the parent directory containing the files and subdirectories from which you want remove old files (i.e., /home/ben/ftp/surveillance). The quotes around this value are not necessary unless the path contains spaces.
  • max_usage is the percent of disk capacity that will trigger the old file deletion actions (i.e., 90 percent).
  • goal_usage is the percent of disk capacity you want to achieve after deleting old files (i.e., 80 percent).

Note that the values of max_usage and goal_usage must be integers.

[ -d "$directory" ] || exit 1
  • Checks that directory exists, otherwise script ends and exits with status 1.
[ "$max_usage" -gt "$goal_usage" ] || exit 1
  • Checks that max_usage is greater than goal_usage, otherwise script ends and exits with status 1.
[ "$( df --output=pcent $directory | \
    grep -Ewo '[0-9]+' )" -ge "$max_usage" ] || exit 0
  • Gets the current disk capacity percent used and checks if it meets or exceeds the threshold set by max_usage. If not, further processing is not required so the script ends and exits with status 0.
dev_used="$( df -B 1K --output=used $directory | \
    grep -Ewo '[0-9]+' )"
  • Gets the currently disk capacity kilobytes used.
goal_usage="$( printf "%.0f" \
    $( echo ".01 * $goal_usage * \
    $( df -B 1K --output=size $directory | \
        grep -Ewo '[0-9]+' )" | bc ) )"
  • Converts the goal_usage variable to kilobytes (we'll need this value further down).
find $directory -type f -printf '%Ts,%k,\047%p\047\n'
  • Locates all files in directory (and in all its subdirectories) and makes a list of these files, one per line, formatted as timestamp, size in kilobytes, 'full/path/to/file'. Note that the 'full/path/to/file' is enclosed in single quotes so spaces in the names of files or directories will not cause problems later.
sort -k1
  • Sorts the previously echo'd list of files by timestamp (oldest first).
awk -F, -v goal="$(($dev_used-$goal_usage))"
  • awk creates an internal variable goal that is equal to the difference between dev_used and goal_usage - and this is the total kilobytes worth of files that must be removed in order to bring the disk capacity percent down to the goal_usage set at the start of the script.
(sum+$2)>goal{printf "%s ",$3; exit} \
(sum+$2)<=goal{printf "%s ",$3}; {sum+=$2}'
  • awk (continued) begins processing the list by keeping a running sum of field 2 values (size in kilobytes) and printing field 3 values ('full/path/to/file') to a space separated string until the sum of kilobytes from field 2 becomes greater than the goal, at which point awk stops processing additional lines.
xargs rm
  • The string of 'full/path/to/file' values from awk is piped to xargs which runs the rm command using the string as its arguments. This removes those files.
1
  • looks like these script can not deal with spaces in subdir or filepath under mainpath :( is there a easy fix for it?
    – p4nda
    Jun 3 at 17:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.