I have a 200k files folder that i want to divide into 25k files folders using cp
or any other cli command that could do the thing
3 Answers
I used this command :
files=(*);
num=$((${#files[@]}/8));
k=1
for ((i=0; i<${#files[@]}; i+=$num)); do
tar cvzf files$k.tgz -- "${files[@]:$i:$num}"
((k++))
done
from this post: How to tar.gz many similar-size files into multiple archives with a size limit shared by Jacob Vlijm. It compress all the files in several .tgz
files so that is good for me. You can change the number to divide, here 8 gaves me 8 compressed files with 25k files inside + 1 compressed file with only 2 files inside. That did the job for me.
You can use the 'find' command with xargs and passing multi command to first read a value from a file and set it to a variable which we will use it in new directory name as an ID and then create that directory next move the files into it and increment the ID counter and save it in a file to keep the last value of it.
here is the command,
find -maxdepth 1 -type f |xargs -n25000 bash -c 'readID=$(< ../ID.txt); echo mkdir directory_$readID; echo mv -v $@ -t directory_$readID/; echo $(expr ${readID} + 1)> ../ID.txt'
P.s. note that please write the start ID in ID.txt file before executing the above command
echo "1" > ../ID.txt
this ID.txt is one level up from current working directory.
once you checked the command and saw what's going on to be happen, then remove 'echo' command from this command 'echo mv -v $@ -t directory_$readID/;
' and behind 'mkdir' to perform actual move, and reset the ID.txt
echo "1" > ../ID.txt
You could tar the directory tree, "split" it, and "cat" it back together.
This might help. Not verified the commands, but I remember it works, from 400 years ago.
https://unix.stackexchange.com/questions/1588/break-a-large-file-into-smaller-pieces
xargs
but i didn't get it to work withfind
and i think this answer could be more suited than the one i find, as my answer just divide files and don't copy a desired number of files