I have a 200k files folder that i want to divide into 25k files folders using cp or any other cli command that could do the thing

  • 1
    related: askubuntu.com/questions/846174/… – Jacob Vlijm Jul 17 '17 at 9:03
  • @AFSHIN Perhap's, can you explicit your answer ? As i searched for man xargs but i didn't get it to work with find and i think this answer could be more suited than the one i find, as my answer just divide files and don't copy a desired number of files – Gy0m Jul 17 '17 at 18:09

I used this command :

for ((i=0; i<${#files[@]}; i+=$num)); do 
tar cvzf files$k.tgz -- "${files[@]:$i:$num}"

from this post: How to tar.gz many similar-size files into multiple archives with a size limit shared by Jacob Vlijm. It compress all the files in several .tgz files so that is good for me. You can change the number to divide, here 8 gaves me 8 compressed files with 25k files inside + 1 compressed file with only 2 files inside. That did the job for me.

| improve this answer | |

You can use the 'find' command with xargs and passing multi command to first read a value from a file and set it to a variable which we will use it in new directory name as an ID and then create that directory next move the files into it and increment the ID counter and save it in a file to keep the last value of it.

here is the command,

find -maxdepth 1 -type f |xargs -n25000 bash -c 'readID=$(< ../ID.txt); echo mkdir directory_$readID; echo mv -v $@ -t directory_$readID/; echo $(expr ${readID} + 1)> ../ID.txt'

P.s. note that please write the start ID in ID.txt file before executing the above command

echo "1" > ../ID.txt

this ID.txt is one level up from current working directory.

once you checked the command and saw what's going on to be happen, then remove 'echo' command from this command 'echo mv -v $@ -t directory_$readID/;' and behind 'mkdir' to perform actual move, and reset the ID.txt

echo "1" > ../ID.txt
| improve this answer | |

You could tar the directory tree, "split" it, and "cat" it back together.

This might help. Not verified the commands, but I remember it works, from 400 years ago.


| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.