Take the 2-minute tour ×
Ask Ubuntu is a question and answer site for Ubuntu users and developers. It's 100% free, no registration required.

So, I have a list of files in a text file. I believe it's about 100,000 files.

The files in said list are spread across many directories, have different sizes, filenames, extensions, ages, etc.

I am trying to find a way to move those files, and just those, to another drive.

Complicating factor: some of the files have the same name, but are not the same file. They can't just be moved into one folder with an overwriting or ignoring policy towards multiples.

Preferably, I would like them to retain their directory structure, but only have the files that I want inside the destination directory. (the destination drive isn't big enough to simply copy everything).

Below is an example of some lines in the text file that has the names of the source files:

media/dave/xdd/cruzer/F#(NTFS 1)/Raw Files/Portable Network Graphic file/3601-3900/FILE3776.PNG/Windows/winsxs/amd64_microsoft-windows-o..disc-style-memories_31bf3856ad364e35_6.1.7600.16385_none_51190840a935f980/Title_mainImage-mask.png
media/dave/xdd/d1/other/hd1/Program Files/DVD Maker/Shared/DvdStyles/Memories/Title_content-background.png

I have tried to use

rsync -av `cat /sourcefile.txt` /media/destinationhdd

which complains that there are too many arguments.

Also

rsync -a --files-from=/sourcefile.txt / /media/destinationhdd

and

cat /sourcefile.txt | xargs -n 200 rsync -av % /media/destinationhdd

However, this just tries to copy my root directory to the destination.


How do I just copy the specific files that I want to?

share|improve this question
    
would you care for a python script? and if so, would one directory be ok of the duplicates (names) would be renamed to: filename_1, filename_2 etc? –  Jacob Vlijm Aug 11 at 19:51
    
I would have suggested --include-from, but is that the same as --files-from, I don't even see that option in the man page. –  dan08 Aug 11 at 19:52
    
Hi Jacob, Yeah, a python script would do me fine. I'd like to know why what I tried didn't work, though. The only thing I can thing of is the text file was unreadable by the command so it went for root instead? That doesn't sound likely, though... Will try the answer below in the meantime and report back. –  Ne Mo Aug 11 at 20:57

3 Answers 3

up vote 2 down vote accepted

Here is a little shell script for you:

#!/bin/sh

while read line
do
    DIR=`dirname "$line"`
    mkdir -p "$2/$DIR"
    mv "$line" "$2/$DIR"
done < $1

Usage (assuming you saved the script as script.sh and made it executable with chmod +x script.sh):

./script.sh input.txt output_directory

It will move all files listed in input.txt to output_directory, using their original paths, for example, for an input.txt has the following list:

test.txt
dir1/test.txt
Another Test/something_else.txt

The files will be moved to:

output_directory/test.txt
output_directory/dir1/test.txt
output_directory/Another Test/something_else.txt

I did some testing before posting this answer, but please make sure you try it on a smaller sample first to confirm it's working as expected!

share|improve this answer
    
Thanks got me closer than ever, but... it seems to copy the directories without copying the files themselves. Error returned is mv: cannot stat ‘/media/dave/xdd/.../waxing-crescent_partly-cloudy.png\r’ Thought at first it might be because of spaces in the filenames, but it happens for every file, even those without any spaces in the name. Then I noticed that it said /r at the end of every filename. Don't see how your script could have produced that, so could be a formatting issue with the list file, will have a look... –  Ne Mo Aug 11 at 21:30
3  
@NeMo Yeah, that's probably the carriage return character. Was the file created/edited in Windows? You should be able to fix it by vi +':set ff=unix' +':wq' sourcefile.txt. –  Alaa Ali Aug 11 at 21:58
    
Thanks, after I changed the lines to be unix lines it FINALLY worked. edit: hmm, maybe not. Some of the lines seem to have retained \R. But some have been changed. Very confusing :/ –  Ne Mo Aug 11 at 23:34

The following script first copies the directory structure from the source, then the files from your list into the corresponding folders. The line if not line in ["", "\n"] is to prevent errors in case the filelist contains empty lines.

#!/usr/bin/env python

import os
import shutil

source = "/path/to/source"; target = "/path/to/target"; filelist = "/path/to/filelist.txt"

for root, dirs, files in os.walk(source):
    for dr in dirs:
        dr = root+"/"+dr
        destdir = dr.replace(source, target)
        if not os.path.exists(destdir):
            os.makedirs(destdir)

with open(filelist) as lines:
    src = lines.readlines()

for line in src:
    if not line in ["", "\n"]:
        shutil.copyfile(line.replace("\n", ""),
            line.replace("\n", "").replace(source, target))

How to use

  • Copy the script into an empty file, save it as move.py
  • Add the appropriate paths in the head section
  • Run it by:

    python /path/to/move.py
    
share|improve this answer

You can do this with xargs fairly easily.

mkdir /newroot/
<filenames.txt xargs -I% bash -c 'mkdir -p /newroot/$(dirname "%" && cp "%" "/newroot/%"'

The tough bit is making sure the new directory structure exists. For that we use dirname to get the directory names, mkdir -p to build them and finally cp (or mv) to copy/move from one to the other. I've left it in copy mode so you can test.

I would suggest testing that find /newroot/ -type f | wc -l and wc -l filenames.txt both give you the same number afterwards.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.