How to move 30 GB of files from a folder containing 80 GB of files?

Question

I have a folder that has more than 500 videos and the folder's size is about 80 GB.

I want to move 30GB of the videos to another folder, but how?

I'm using terminal and can't install a GUI.

Answer 1

Unselectively(automatically)

I don't know of a single command-line tool that can do this ... But, you can use a script to move files until a certain total size has been moved e.g. like this(Run the script only once ... If you run it again it will move the specified size again from the remaining files):

#!/bin/bash

sourc_dir="/path/to/directory" # Set the path to the source directory "no trailing /"
dest_dir="/path/to/directory" # Set the path to the destination directory "no trailing /"
move_size="30" # Set the total size of files to move in Gigabytes
 
# Don't edit below this line
move_size=$(( move_size * 1000000 ))
current_size=$(du -sk "$sourc_dir" | awk '{printf "%d", $1}')
remaining_size=$(( current_size - move_size ))

for f in "$sourc_dir"/*; do
  [ -f "$f" ] || continue
  new_size=$(du -sk "$sourc_dir" | awk '{printf "%d", $1}')
  (( new_size <= remaining_size )) && exit
  mv -n -- "$f" "$dest_dir"/
  done

Or even a one-liner(no script-file needed) that you can run in the terminal e.g. like so:

for f in /path/to/source/*; do [ -f "$f" ] || continue; (( size >= (30*(1000**3)) )) && break || (( size += $(stat --printf "%s" "$f") ))  && mv -n -- "$f" /path/to/destination/; done

You might want to unset size to be able to rerun it in the same shell instance.

Selectively(manually)

You can always specify a shell pattern or pipe from a search tool based on filenames, sizes, types ... etc.

Or, more conveniently, you can use a console-based file-manager that supports multiple file selection like e.g. the well known and feature rich GNU Midnight Commander that is available from the Ubuntu official repositories and can be installed like so:

sudo apt install mc

and to run it type mc and hit Enter ... You'll then be presented with a navigable UI where you can use the Insert keyboard key to select multiple files ... You will be able to see the total size of the selected files and their count in real time like so:

Once you have finished selecting all the files you want to move, hit the F6 keyboard key to open the move/rename dialog and enter the destination directory then choose [< OK >] to move the selected files at once like so:

Or even the less known with less features fff (Fucking Fast File-Manager) :-) ... That is interestingly written in bash ... Although it supports multiple file selection, you might need to customize it a bit to be able to see total selected files size e.g. as a quick patch, you can change line 159 in the source fff file from this:

local mark_ui="[${#marked_files[@]}] selected (${file_program[*]}) [p] ->"

to this:

local mark_ui="[${#marked_files[@]}] $(for f in "${marked_files[@]}"; do (( tsize += $(stat --printf "%s" "$f") )); done; tsize=$((tsize/(1000**2))); printf '[%d M]' "$tsize";) selected (${file_program[*]}) [p] ->"

Then you can simply run that file like so(No installation needed):

bash fff

Or view help on usage like so:

bash fff -h

Then all you have to do is simply navigate to your source directory and start selecting files to move with the m(lower case) keyboard key like so:

When finished selecting files, navigate to the destination directory and move the previously selected files to it by hitting the p(lower case) keyboard key.

Answer 2

Here's an alternative to Raffa's idea that does not require a du on the whole directory for every file. Instead it does a stat on each file once, using find's -printf %s to get the size and %p to get the path to the file to be moved.

#!/bin/bash

src='foo'
dest='bar'

# 30GB in bytes to move:
(( bytestomove=30*(1000**3) ))
movedbytes=0

while IFS=$'\t' read -r -d '' bytes file
do
    # if we already moved enough, break out
    (( movedbytes >= bytestomove )) && break;

    # try to move it, break out if it fails
    mv -n -- "$file" "$dest" || break

    # successfully moved, add bytes to movedbytes
    (( movedbytes += bytes ))
done < <(find "$src" -type f -printf "%s\t%p\0")

(( movedmegs=movedbytes/(1000**2) ))
printf "Moved %d MB\n" $movedmegs

Answer 3

@TedLyngmo pointed out that the tar solution I first presented effectively copies the files, and doesn't move them. This distinction is important for 2 reasons:

1. With a copy, the source files remain, whereas with a move, the source files no longer exist in their source locations.
2. If files are moved within the same filesystem, as well may be the case in this question, mv will simply relink the source file into the new location and not copy the content at all. This will typically be significantly faster than copy, especially for large files.

Move Scenario

We can still use the tar concepts in the copy scenario originally given to build a file list containing just the first 30GB of files that tar finds, and then move just those files:

#!/bin/bash

# Convert human-readable number, e.g. 1M, 30G to number of 1024 byte blocks 
limit=$(numfmt --from=auto $1 --to-unit 1024)
src_dir=$2
dest_dir=$3

mkdir $dest_dir
filelist=$(readlink -f $(mktemp -p $dest_dir))

# create a list of files to be moved.  tar -v outputs filenames to stdout, and 
# the dest archive is /dev/null
tar -v -L $limit -F false -c -f /dev/null -C $src_dir . > $filelist 2> /dev/null
tar_result=$?
if [ "$tar_result" = 2 ]; then
    echo "$1 limit was hit - some unmoved files will remain in $dest_dir"
else
    echo "All files will be moved"
fi

pushd $dest_dir > /dev/null
abs_dest_dir=$PWD
# create directories in the destination
grep '/.*/$' $filelist | xargs mkdir -p
popd > /dev/null

pushd $src_dir > /dev/null
# create hard links in the destination
grep -v '/$' $filelist | xargs cp -l --parents -t $abs_dest_dir
# unlink source files
grep -v '/$' $filelist | xargs rm
# remove source directories in reverse order, if they are empty
grep '/.*/$' $filelist | tac | xargs rmdir --ignore-fail-on-non-empty
popd > /dev/null

rm $filelist

This is more complex than I'd like:

• mv doesn't give as much control of source and dest paths for multiple files (specifically no analog of cp's --parents option), so the file moves are achieved by creating hard links in the destination locations, then unlinking the source files

• Because source directories contain a mix of entries that are and are not to be moved, directories must be recreated in the destination, and not simply moved (along with all contents). So the appropriate destination directories are created before moving files, and corresponding source directories are removed after moving files, if they are empty.

Caveats

• tar -v will output one file entry per line. This will fail spectacularly if there are special characters in the pathnames, e.g. newlines. Hopefully source path names are sensibly named.

• This will move a bit less than 30GB of actual file data, as the stream contains meta-data (filenames, timestamps, etc) as well as the actual contents, and this is all subject to the 30GB limit.

• I'm playing a bit fast-and-loose with KB vs. kb here. Since the -L parameter works on units of 1024 bytes, this is actually limiting move to 3,072,000,000 bytes of tar stream. Adjust the numbers as appropriate.

• filelist.txt is created in the current directory. Write permissions will be required for this, this is probably a given, since the move operations will require write permissions anyway.

Copy Scenario

You can use tar to turn your folder into a stream, then tar again to turn the stream back into the files in a new location. The first tar uses -L to limit the "tape-length" to 3000000 KB, before asking for a new tape. tar uses -F to run false after each "tape". Because false returns an error, transfer stops after the the first "tape", i.e. 30GB. Having all data go through just 2 invocations of tar should be faster than looping over all files with bash loops.

mkdir dest.dir
tar -cv -L 3000000 -F false -C src.dir . | tar -x -C dest.dir/

This will copy the files until the 30GB limit is hit, then the tars will error out with:

tar: Unexpected EOF in archive
tar: Unexpected EOF in archive
tar: Error is not recoverable: exiting now
tar: ‘false’ command failed
tar: Error is not recoverable: exiting now

tar will not transfer the file that was being transferred when the limit was hit - it will not be written to the destination at all.

Caveats

• This will copy a bit less than 30GB of actual file data, as the stream contains meta-data (filenames, timestamps, etc) as well as the actual contents, and this is all subject to the 30GB limit.

• I'm playing a bit fast-and-loose with KB vs. kb here. Since the -L parameter works on units of 1024 bytes, this is actually limiting transfer to 3,072,000,000 bytes of tar stream. Adjust the numbers as appropriate.

---

Previous edits of this answer used find-cpio-head-cpio, and then tar-head-tar. The tar-tar solution now presented should be sufficient, but please inspect the edit history if the other solutions are of interest.

Answer 4

Go to the folder where your files are:

cd SOURCE_FOLDER

make a list of files in that folder

ls > list.txt

Edit it in any editor and remove files you do NOT want to move. Save the result so now list.txt has only the files you want to move. Now run this simple script that reads each line from your list.txt file and moves that file to your destination directory.

while IFS= read -r line; do mv "$line" ../DESTINATION_DIRECTORY/"$line"; done < list.txt