3

I have nearly 4 lakhs (400,000) of files in a folder named model and from this folder I have selected a few thousand files and their names are saved in id.txt. Now I want to take only the files listed in id.txt from the folder model and copy them to another folder named selected_ids.

The two folders are present inside the same directory test which contains

  1. model
  2. selected_ids
  3. id.txt

id.txt contains filenames one by one eg:

ENSP00000290866.4_2.pdb.xz
385719215_2.pdb.xz
GENSCAN00000006392_3.pdb.xz

Model folder contains nearly 3 lakh files

ENSP00000290866.4_2.pdb.xz
385719215_2.pdb.xz
GENSCAN00000006392_3.pdb

*note: in folder some files have .pdb.xz extension and some only .pdb

I know how to copy a single file from one folder to another like this

cp model/ENSP00000290866.4_2.pdb.xz selected_ids/

But I have thousands and thousands of files to copy in short time.

Can anyone give some efficient command to copy the files?

4

the command you are looking is as follow,

cd /path/to/model
$ xargs -d '\n' -a /path/to/id.txt cp -t /path/to/selected_ids

this will read the files name saved in id.txt and copy to selected_ids directory.

  • @ AFSHIN linux[model_1]xargs -d '\n' -a ../id_original.txt cp -t ../selected_ids I am getting following error continuous: cp: cannot stat `GENSCAN00000051451_3.pdb.xz\r': No such file or directory .................... But when i checked in the model folder it shows that particular file is present, find . -name GENSCAN00000051451_3.pdb.xz ./GENSCAN00000051451_3.pdb.xz I observe that some files are able to read, when i took 30 ids and ran the script its working but not for whole file. In folder model_1 there are some files named .pdb and .pdb.xz also. – Nitha Jul 20 '17 at 12:26
  • Filtering step performed for each .pdb structures in excel and filtered out best model. Only their id name was saved as id.txt – Nitha Jul 20 '17 at 19:15
  • I have changed and ran the comment, but still it shows error. xargs -d '\n' -a ../id_original.txt cp -t ../selected_models error: cp: cannot stat ENSP00000474609.1_2.pdb.xz\r': No such file or directory xargs -d '\n' -a ../id_original.txt cp -t ../selected_models error: cp: cannot stat \nGENSCAN00000001365_1.pdb.xz': No such file or directory – Nitha Jul 21 '17 at 4:19
  • No.. the files are present inside the model folder GENSCAN00000001365_1.pdb.xz @bio-linux[model_1] find . -name GENSCAN00000001365_1.pdb.xz [ 9:47AM] ./GENSCAN00000001365_1.pdb.xz – Nitha Jul 21 '17 at 4:38
  • show me your id.txt content run cat -eT id.txt, you can paste here paste.ubuntu.com and share the link – αғsнιη Jul 21 '17 at 5:19
0

Use while IFS= read -r variable; do... done < input.txt structure to read file line by line and execute desired code in ... part. This is a standard bash way to read file line-by-line and act upon it. With the directory as you described, it should be done as so:

$ tree                                                                                                                                               
.
├── 385719215_2.pdb.xz
├── ENSP00000290866.4_2.pdb.xz
├── GENSCAN00000006392_3.pdb.xz
├── id.txt
├── model
└── selected_ids

2 directories, 4 files
$ while IFS= read -r file; do cp "$file" selected_ids/"$file" ;done < id.txt                                                                         
$ tree
.
├── 385719215_2.pdb.xz
├── ENSP00000290866.4_2.pdb.xz
├── GENSCAN00000006392_3.pdb.xz
├── id.txt
├── model
└── selected_ids
    ├── 385719215_2.pdb.xz
    ├── ENSP00000290866.4_2.pdb.xz
    └── GENSCAN00000006392_3.pdb.xz
  • Ubuntu 17.04: The program 'tree' is currently not installed. You can install it by typing: sudo apt install tree – madneon Jul 28 '17 at 21:26
  • @madneon Yeah, it's not a default program, but tree isn't important as far as the question goes, it's used here only for giving visual idea of how directory is structured. The important part is the while loop – Sergiy Kolodyazhnyy Jul 28 '17 at 21:36
0

Assuming there is no white-spaces in input file you can simply use for loop:

for f in $(<id.txt); do cp -v "model/$f" selected_ids/;done

If you having trouble with "\r" at the end of file names (most likely file was saved in Windows-family OS), you also need to trim variable $f:

for f in $(<id.txt); do f=`echo $f|tr -d "\n\r"`; cp -v "model/$f" selected_ids/; done

Note that cp with option -v will display the current file being copied, so its easy to verify this batch copying process.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.