how to move a specific range of files from one directory to another

Question

In my parent directory I have 10000 files, I want to copy or move the files to 4 sub directories subdir1, subdir2, subdir3, subdir4 equally and if possible simultaneously. Is to possible to copy a certain range of files from the parent directory to the sub-directories i.e.,

1-2500 files to subdir1
2500-5000 files to subdir2
5000-7500 files to subdir3
7500-10000 files to subdir4

All of these should be done using a batch file. Is it possible? Pls help me out if any one knows.

Thanks in Advance.

Answer 1

This one works for an arbitrary number of files and can deal with weird file names (those containing spaces, newlines, backslashes or other weirdness):

#!/usr/bin/env bash

## This will keep track of the number of files processed
num=0;
## This is used to choose the righht subdir
dir=1;
## The initial value of the target directory
target="subdir1"

for file in *; do 
    ## Skip unless this is a file
    if [ -f "$file" ]; then
        ## Create the target directory if it doesn't exist
        [ -d "$target" ] || mkdir  "$target"
        ## Move the current file
        mv "$file" "$target"
        ## Change the target if we're at a multiple of 2500
        if [[ $(( ++num % 2500 )) -eq 0 ]]; then
            target="subdir"$((++dir));
        fi
    fi
done

You can also implement the same thing using find:

#!/usr/bin/env bash

## This will keep track of the number of files processed
num=0;
## This is used to choose the right subdir
dir=1;
## The initial value of the target directory
target="subdir1"

## Run your find, with -print0 to print NUL-separated values. This
## is needed for file names that contain newlines
find . -type f -print0 |
    ## The IFS= makes this deal with spaces, the -d ''  sets the input delimiter
    ## to NUL so ti can work with -print0 and the -r makes it work with backslashes
    while IFS= read -d '' -r file; do
    ## Create the target directory if it doesn't exist
    [ -d "$target" ] || mkdir  "$target"
    ## Move the current file
    mv "$file" "$target"
    ## Change the target if we're at a multiple of 2500
    if [[ $(( ++num % 2500 )) -eq 0 ]]; then
        target="subdir"$((++dir));
    fi
    done             

Save that script as ~/bin/batch_rename.sh, make it executable (chmod a+x ~/bin/batch_rename.sh) and then run it from the directory where the files are.

---

NOTES

• The first example will only find files in the current directory. To make it recursive, add this line to the beginning:

  shopt -s globstar
  

  Then, change the for file in * to for file in **/*.

• The second example will find all files in this and any subdirectory. That may or may not be what you want.

Answer 2

If the order is not an issue, the script below:

• splits the files into (arbitrary) chunks
• creates the sub-directories per chunk (chunk_1, chunk_2 etc.)
• moves the corresponding files into the sub-directories

Note, that:

• If the order is an issue, the script needs minor adjustment, but then please include the order- rules in the question.
• The script does not "mind" files (-names) with spaces etc. Even the subdirectory's name "body" may include spaces.

The script

#!/usr/bin/env python3
import os
import shutil
import sys
#--- if desired, change the sub dir's name body below
namebody = "chunk_"
#---
dr = sys.argv[1]; size = int(sys.argv[2]); 
files = [f for f in os.listdir(dr) if os.path.isfile(dr+"/"+f)]

n = max(1, size)
chunks = [files[i:i + size] for i in range(0, len(files), size)]
for i, item in enumerate(chunks):
    subfolder = os.path.join(dr, namebody+str(i+1))
    if not os.path.exists(subfolder):
        os.makedirs(subfolder)
    for f in chunks[i]:
        shutil.move(dr+"/"+f, subfolder+"/"+f)

How to use

1. Copy the script into an empty file, save it as reorganize.py

2. if desired, you can change the sub directory's name "body" (the name section without the number) in the head of the script, in:

   namebody = "chunk_"
   

3. Run it with the main directory and the chunk size as arguments:

   python3 /path/to/reorganize.py <main_directory> <chunk_size>
   

Answer 3

Based on my old script with some modification (just changed the N value and variable names :):

Grouping each N files in separate directories

##grouping each N files in separate directories

echo 1 > dircount; 
find source -type f -name 'filterFileName' -print0 | sort -z --version-sort | \
  xargs -0n2500 bash -c 'read TARGET <dircount; \
   echo mkdir -p "subdir$TARGET"; \
   echo mv -t "subdir$TARGET" "$@"; \
  expr $TARGET + 1 >dircount' grouping

1. -print0 prints the filenames separated by the NUL (\0) character. It's the safest way to pass filenames as output to other commands.
2. sort with -z looks for nul-delimited input, and the --version-sort allows to safely sort variable length numbers, so that filename2.xyz comes before fileName3.xyz
3. xargs with -n2500 limit the number of arguments applied to each command (in this case 2500 arguments). The -0 is for nul-delimited input.

Note: Don't forget that you are running the script in test case, so clear the echo command next to the related lines to performing to run actual script.

---

Also there are two another scripts which are not as fast as above one:

groupFiles=0
TARGET=1
for file in `ls -v /path/to/source/filterFileName` ; do
    mkdir -p "subdir$TARGET" && mv "$file" "subdir$TARGET" 
    [[ ++groupFiles -eq 2500 ]] && groupFiles=0 &&  ((TARGET++))
done

• ls -v command sorts the files as natural sort of (version) numbers.
  You can parse the ls command if files name has no whitespace, new-line, etc.

• mkdir -p "subdir$TARGET" crates the directory based on TARGET variable.

• mv "$file" "subdir$TARGET" moves the file to directory which TARGET specifies.

• Reset the groupFiles=0 to zero when 2500 files has been moved ([[ ++groupFiles -eq 2500 ]]) and increment to TARGET value.

Note that change /path/to/source/ to your actual source directory.

If you don't want to parse the ls command due files name including spaces, new-lines and etc, here is another option to do:

groupFiles=0
TARGET=1
find /path/to/source/ -type f -name 'filterFileName' -print0 | \
     sort -z --version-sort | while IFS= read -d '' -r file; do \
     mkdir -p "subdir$TARGET" && mv "$file" "subdir$TARGET" ;
     [[ ++groupFiles-eq 7 ]] && groupFiles=0 &&  ((TARGET++));
done

Answer 4

This bash script will move any number of files present in the target directory passed to it as an argument subdiving them evenly across any number passed to it as an argument of target subdirectories named subdir<N>, creating the target subdirectories if not present already; it should be placed outside of the target directory to be run against to avoid it being moved during the execution.

Usage:

./script.sh <path_to_target_directory> <number_of_subdirectories>

_{*<path_to_target_directory> = path to the directory containing the files to subdivide; <number_of_subdirectories> = number of subdirectories in which to subdivide the files}

#!/bin/bash

for i in `seq 1 "${2}"`
do
    mkdir -p \'"${1}"/subdir"${i}"\'
done
j=0
find \'"${1}"\' -maxdepth 1 -type f | while read -r filepath
do
    N=$(( ${j} % ${2} + 1 ))
    mv \'"${filepath}"\' \'"${1}/subdir${N}"\'
    ((${j}++))
done

Results for ./script.sh ~/testdir 4:

Before:

~/testdir
├── 1
├── 10
├── 2
├── 3
├── 4
├── 5
├── 6
├── 7
├── 8
└── 9

After:

~/testdir
├── subdir1
│   ├── 1
│   ├── 2
│   └── 7
├── subdir2
│   ├── 10
│   ├── 6
│   └── 8
├── subdir3
│   ├── 3
│   └── 9
└── subdir4
    ├── 4
    └── 5