3

I have a 200k files folder that i want to divide into 25k files folders using cp or any other cli command that could do the thing

2
  • 1
    related: askubuntu.com/questions/846174/… Jul 17, 2017 at 9:03
  • @AFSHIN Perhap's, can you explicit your answer ? As i searched for man xargs but i didn't get it to work with find and i think this answer could be more suited than the one i find, as my answer just divide files and don't copy a desired number of files
    – Gy0m
    Jul 17, 2017 at 18:09

3 Answers 3

1

I used this command :

files=(*); 
num=$((${#files[@]}/8));
k=1
for ((i=0; i<${#files[@]}; i+=$num)); do 
tar cvzf files$k.tgz -- "${files[@]:$i:$num}"
((k++))
done

from this post: How to tar.gz many similar-size files into multiple archives with a size limit shared by Jacob Vlijm. It compress all the files in several .tgz files so that is good for me. You can change the number to divide, here 8 gaves me 8 compressed files with 25k files inside + 1 compressed file with only 2 files inside. That did the job for me.

0

You can use the 'find' command with xargs and passing multi command to first read a value from a file and set it to a variable which we will use it in new directory name as an ID and then create that directory next move the files into it and increment the ID counter and save it in a file to keep the last value of it.

here is the command,

find -maxdepth 1 -type f |xargs -n25000 bash -c 'readID=$(< ../ID.txt); echo mkdir directory_$readID; echo mv -v $@ -t directory_$readID/; echo $(expr ${readID} + 1)> ../ID.txt'

P.s. note that please write the start ID in ID.txt file before executing the above command

echo "1" > ../ID.txt

this ID.txt is one level up from current working directory.

once you checked the command and saw what's going on to be happen, then remove 'echo' command from this command 'echo mv -v $@ -t directory_$readID/;' and behind 'mkdir' to perform actual move, and reset the ID.txt

echo "1" > ../ID.txt
0

You could tar the directory tree, "split" it, and "cat" it back together.

This might help. Not verified the commands, but I remember it works, from 400 years ago.

https://unix.stackexchange.com/questions/1588/break-a-large-file-into-smaller-pieces

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .