Script to delete old files(tar) in a directory if the number exceeds 15?

Question

Please anyone can give the solution for this I'm trying to take gitlab backup to another server, script should delete the old files if the file number in the directory exceeds 15?

Answer 1

Diclaimer:I've tested the following commands with filenames that contain spaces, but not with filenames that contain newlines. I suspect that they will not play nicely with filenames containing the newline character, and would avoid using them if you suspect filenames containing them may be created.

This approach relies on the ctime of the file so if files have any of their attributes changed, they will appear to be newer than their creation time. Only you can decide whether relying on ctime is applicable in your situation. If you'd rather use mtime, change -printf "%C+ %p\n" to -printf "%T+ %p\n" in the find commands.

The following command can be issued inside the directory containing your tar files. This assumes that the filenames are all something like something.tar. If the filenames are not of this format, the command will need to be modified, either changing the -iname '*.tar.' to -iname '*.tar.gz*' if the files are .tar.gz files, or by removing the entire -iname '*.tar.' string if you just want to operate on any files in that directory, regardless of filename format.

find . -mindepth 1 -maxdepth 1 -type f -iname '*.tar' -printf "%C+ %p\n" | sort -n | cut -d ' ' -f 2- | head -n -15 | xargs -I{} echo "{}"

If this shows you the oldest files, outside of your 15 file limit, then use the following command to delete those files.

find . -mindepth 1 -maxdepth 1 -type f -iname '*.tar' -printf "%C+ %p\n" | sort -n | cut -d ' ' -f 2- | head -n -15 | xargs -I{} rm "{}"

By Way of explanation:

• find . -mindepth 1 -maxdepth 1 -type f -iname '*.tar' -printf "%C+ %p\n" will list all of the files that end in .tar in . (the current directory) without recursing into subdirectories. It then prints the ctime timastamp followed by a space and the file name with a trailing newline character.
• | sort -n sorts the output of find numerically, so files are listed from oldest to newest (by ctime) .
• | cut -d ' ' -f 2- removes the added timestamp that find created, but preserves the order of the files listed by sort.
• | head -n -15 trims the bottom 15 items from the output of cut.
• xargs -I{} rm "{}" runs the rm command on each file, ensuring the filename is not split on whitespace.

This can be written as a bash script, with the number of files to retain and the directory on which to operate as variables in the script. It's possible to pass in the directory and file retention count as arguments instead, but I won't cover that here.

#!/bin/bash

file_limit=15
dir=/directory/containing/tarfiles

find "$dir" -mindepth 1 -maxdepth 1 -type f -iname '*.tar' -printf "%C+ %p\n" | sort -n | cut -d ' ' -f 2- | head -n -"$file_limit" | xargs -I{} rm "{}"

If you save this script somewhere e.g. /home/user/trim_old_gits and ensure that you have given it executable permissions. The script can be run from the command line by entering:

/home/user/trim_old_gits

Or from within /home/user:

./trim_old_gits

As mentioned in Jacob's marvellous python answer, using the cron utility would be a good way to ensure that this happens on a regular basis, if it's not crucial that the files be deleted immediately, or inotifywait if timing is more sensitive.

Answer 2

Given the fact that both ctime and mtime are no guarantee that you actually delete the oldest files, depending on what happened to the files in between, the script below deletes the files, exceeding an arbitrary number, inside a given directory.

(a.o.) here we can read:

ctime is the inode or file change time. The ctime gets updated when the file attributes are changed, like changing the owner, changing the permission or moving the file to an other filesystem but will also be updated when you modify a file. mtime

mtime is the file modify time. The mtime gets updated when you modify a file. Whenever you update content of a file or save a file the mtime gets updated.

Most of the times ctime and mtime will be the same, unless only the file attributes are updated. In that case only the ctime gets updated.

Having said that

According to the ctime of a file, the tiny background script below will delete the oldest files if the number of files exceeds a set number. It is yours to decide if that is a usable option in your situation.

The script

#!/usr/bin/env python3
import sys
import os
from operator import itemgetter
import time

dr = sys.argv[1]; n = int(sys.argv[2])

while True:
    time.sleep(3)
    # list files; get the number of files
    files = [os.path.join(dr, f) for f in os.listdir(dr)]; nfiles = len(files)
    if nfiles > n:
        # if nfiles exceeds the threshold, get the number of files to delete
        todel = nfiles - n
        # sort the list by creation date, delete the oldest
        del_list = sorted([
            [f, os.path.getctime(f)] for f in files
            ], key=itemgetter(1))[:todel]
        for f in del_list:
            os.remove(f[0])

How to use

1. Copy the script into an empty file, save it as keep_latest.py

2. Test- run the script from a terminal with the path to your directory and the number of (latest) files to keep as arguments:

   python3 '/path/to/keep_latest.py' '/path/to/directory' 15
   

   to keep the latest 15 files in '/path/to/directory'

3. If all works fine, add to Startup Applications: Dash > Startup Applications > Add. Add the command:

   python3 '/path/to/keep_latest.py' '/path/to/directory' 15
   

Other options

The script above is one of the many options. If either mtime or ctime would suffice, another option would be to use inotifywait, and make it do the same as the script above, but only if a file is added-, moved- or copied into the directory.

If time accuracy (immediate removal of extra files) is not really important, also a command, run by cron would be a good option.

If either the loop of inotifywait or the script above would be more efficient would be object of testing and comparing.

Either way, the used resources would be practically none.

Answer 3

bash script

• list files
• if more then limit => start deleting the files until limit reached

# create array of files sorted on timestamp (oldest first)
rr=$(find /data/*.tgz -type f -printf '%T+ %p\n' | sort)

# if more then 7 tgz files, list the oldest files to array files_to_delete
files_to_delete=$(echo -e "$rr"| awk -v ftd="$(($(echo -e "$rr"| wc -l) - 7))" '{if(ftd<=0) exit; print $NF; if(NR==ftd)exit;}')

# delete the files
echo -e "$file_to_delete" | xargs rm -vf