4

I'm creating a timelapse video by splitting a video into individual images, using G'MIC via the command line to average every 7 frames, then output the results to new images which will result in the frames for the timelapse (this way I get noise free video that looks great). As you can imagine I have a lot of frames - several thousand individual images in a directory, named sequentially (image1.tiff, image2.tiff, etc.). I found a script that works perfectly for moving the files - once...

k=1; find source/ -type f | while read file; do
     [[ k++ -le 7 ]] && mv "$file" target/ 
done 

What I need now is to make this script

  1. repeat itself until all the files are moved and
  2. create sequentially numbered target directories (1, 2, 3, etc.) as it goes so I end up with several hundred directories each with 7 images inside.

Just to clarify, directory 1 should contain images 1-7, directory 2 should contain images 8-14, etc.

Any help is greatly appreciated, I'm pretty much stuck with this one.

  • You guys are awesome. Sorry for the late response, it's been a busy few days. Several of these work perfectly, however I've marked @Jacob-Vlijm's answer as most helpful simply because it worked precisely the way I envisioned this to work. However I'm keeping all of these as I reckon I can learn something from everyone's contribution. Thanks again and Happy New Year! – Tim Jan 2 '15 at 4:49
3

The (python) script below creates sub folders and organizes your files into the folders:

Additionally:

  • The script calculates the number of folders and adds leading zeros to the folder names for proper sorting (since you mentioned thousands of files, meaning hundreds of folders)
  • The number of files per folder may vary, depending on your setting in the head of the script

The script assumes:

  • All files have an extension, all files have the same extension
  • Your files are named image1.tiff, image2.tiff etc (no leading zeros).

The script

#!/usr/bin/env python3
import os
import math
import shutil
#---
directory = "/path/to/files"      # path to your files
n_perfolder = 7                   # number of files per sub folder
#--
# creating file list, extension
f_list = os.listdir(directory); ext = f_list[0].split(".")[-1]
# calculate number of folders
n_folders = math.ceil(len(f_list)/n_perfolder)
# creating folder names, including leading zeros
folders = [str(fn+1) for fn in range(n_folders)]
fl_names = [(len(max(folders, key=len))-len(fl))*"0"+fl for fl in folders]
# creating folders and allocate files 
for i in range(len(fl_names)):
    mkfolder = directory+"/"+fl_names[i]
    if not os.path.exists(mkfolder):
        os.makedirs(mkfolder)
        r = range(i*n_perfolder+1, i*n_perfolder+n_perfolder+1)
        for n in r:
            try:
                file = directory+"/"+"image"+str(n)+"."+ext
                target = mkfolder+"/"+"image"+str(n)+"."+ext
                shutil.copyfile(file, target)
            except FileNotFoundError:
                pass

How to use

Copy the script into an empty file, in the head section, set the directory to your files and the number of files per sub directory, save it as organize.py.

Run it by the command:

python3 /path/to/organize.py

Note

If you'd like to move the files instead of copying, replace the line:

shutil.copyfile(file, target)

by:

shutil.move(file, target)

(mind the indent!)

2
sevenfile=0
newdir=1
for file in `ls -v /path/to/source/image*.tiff` ; do
    mkdir -p "$newdir" && mv "$file" "$newdir" 
    [[ ++sevenfile -eq 7 ]] && sevenfile=0 &&  ((newdir++))
done
  • ls -v command sorts the files as natural sort of (version) numbers.
    You can parse the ls command in this your case, because all the files are specific and only are image*.tiff.
  • mkdir -p $newdir crates the directory based on newdir variable.

  • mv "$file" "$newdir" moves the file to directory which newdir specifies.

  • Reset the sevenfile=0 to zero when 7 files has been moved ([[ ++sevenfile -eq 7 ]]) and increment to newdir value.

Note that change /path/to/source/ to your actual source directory.

If you don't want parsing the ls command and if files name including spaces, newlines and etc:

find /path/to/source/ -type f -name 'image*.tiff' -print0 | \
     sort -z --version-sort | while IFS= read -d '' -r file; do \
     mkdir -p "$newdir" && mv "$file" "$newdir" ;
     [[ ++sevenfile -eq 7 ]] && sevenfile=0 &&  ((newdir++));
done
2

If you don't mind everything being moved separately:

#! /bin/bash
for ((i = 1; ; i++))
do
    mkdir -p target-$i
    for j in $(seq $((i*7 - 6)) $((i*7)))
    do
        mv -t "target-$i" "source/image$j.tiff" || exit 1
    done
done

Or, more concise:

find source -type f -name 'image*.tiff' -print0 | \
  sort -z --version-sort | \
  xargs -0n7 bash -c 'TARGET=target-$((${1//[^0-9]/} / 7 + 1)); \
    echo mkdir -p "$TARGET"; \
    echo mv -t "$TARGET" "$@"' move-7

Explanation:

  1. -print0 prints the filenames separated by the NUL (\0) character. It's the safest way to pass filenames as output to other commands.
  2. sort with -z looks for nul-delimited input, and the version sort allows us to safely sort variable length numbers, so that image2.tiff comes before image119.tiff
  3. xargs can limit the number of arguments applied to each command with -n, so here we use -n7. The -0 is for nul-delimited input.
  4. We extract the quotient of the first argument's number $((${1//[^0-9]/} / 7 + 1)) - ${1//[^0-9]/} deletes everything in the argument which is not a number. Integer division ensures we get the quotient.
  5. This should be faster, since we run exactly the number of mvs required.

Inspect the commands produced, and if they look okay, run it again after removing the echos.

KasiyA has an interesting modification to this.


JJoao's answer brings rename to mind. First, make all the directories:

FILES=(source/image*.tiff)
COUNT=${#FILES[@]}
DIR_NUM=$((COUNT / 7 + 1))
seq 1 $DIR_NUM | xargs bash -c 'mkdir -p "${@/#/target-}"'

rename -n 's;source/image(\d+)\.tiff$;"target-".int($1 / 7 - 1)."/image".$1.".tiff";e' source/image*.tiff

In fact, we could apply the first four lines to all solutions here, and skip the repeated mkdir -p.

  • What does the move-7 mean on the end? – αғsнιη Dec 31 '14 at 8:35
  • @KasiyA bash -c takes as first argument the value for $0. It isn't directly used, but error messages will use that argument. – muru Dec 31 '14 at 8:55
  • @KasiyA see unix.stackexchange.com/q/152391/70524 – muru Dec 31 '14 at 8:58
  • Good info. and a suggestion instead of TARGET=target-$((${1//[^0-9]/} / 7 + 1)) – αғsнιη Dec 31 '14 at 10:09
1
#!/usr/bin/perl

open(F,"-|","ls source/*") or die;

while(<F>){ chomp;
  if(/(\d+).tiff/){
     $dir= "target-" . (1+int($1/7));
     mkdir($dir) unless -d $dir;
     system "mv $_ $dir";
  }
}
0

You can use xargs (-n or --max-args) to set the number of files to move, but keep in mind that arguments is limited to ARG_MAX.

#!/bin/bash

images="./source/image*.tiff"
target="./target"

ls -v $images | xargs -n 7 | while read -a files; do mkdir $target/$((++n)) && mv -t $target/$n ${files[@]}; done

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.