2

What I'm really tring to do, it verify the the data on an external drive is still readable. I thought the simplest and surest way to do this would be to copy everything to /dev/null. It works great for a file. But when I add -R to recursive copy a dirfectory structure, if fails (presumably) because it can't create subdirectories in /dev/null.

When I try :

cp -R Flintstones/ /dev/null

I get :

cp: cannot overwrite non-directory '/dev/null' with directory 'Flintstones/'

But this works:

cd Flintstones
cp Fred.txt /dev/null

How can I copy the entire directory in one go??

Thank you, and please be gentle with the newbie.

1

Hello and welcome to Ask Ubuntu!

If I understood your question correctly you just want to run some command that will read all files in the given directory, but you are not interested in any output other than potential error messages saying files could not be read. You can achieve that for example using the grep command, like this:

grep -r Zaphod Flintstones/

The above command means that grep will recursively go through all files in the Flintstones directory looking for the string "Zaphod" (you could put some other string there, it does not matter, just something for grep to search for, if you want no output pick something that you think will not occur in any file). If the command completes without error messages about unreadable files, then that means the directory is readable.

Not saying this is the best way, probably there are more elegant ways, but the above is one way of doing it.

| improve this answer | |
  • Bingo!!! That did it. [~] # cd "/share/external/DEV3312_1" [/share/external/DEV3312_1] # ls Remote Backup Seven/ [/share/external/DEV3312_1] # grep -r Zaphod "Remote Backup Seven/" And, sure, there may be some elegant syntax to take each file name in the directory and pipe it to the copy to null command. But this is all this newly-hatched penguin can handle right now. Thank you so much for your help. – pastorbadger Oct 28 '19 at 20:25
  • "cat" and "cp" are not going to work. These commands seem smart enough to realize that since I'm copying to /dev/null, there's no point in actually reading the data. At least, that's what the timings suggest. the "grep" method does work, but only one or two at a time, as they eat the processor. The real cat's pajamas is the "dd" command. As a bonus, it also reads the empty space on the disk (partition?). I can run eight at a time on the 4TB USB drives and processor hovers around 15%. So, "fdisk -l" to find the drive names, and "dd if=/dev/sdm1 of=/dev/null bs=100M" on each drive. Thanks – pastorbadger Nov 8 '19 at 5:47
0

Use cat instead of cp

Instead of using cp to read every file, you can use the cat command.

The screen below illustrates catall.sh bash script. The first section shows results for a short run (40 minutes!) on /home/rick/. The second section shows in progress display on /:

catall progress display.gif

On my system there are many partitions (including two full Windows 10 installations) where all the files are being read.

catall.sh bash script

The bash script must be called with sudo powers because there are files in Linux a regular user cannot read. Additionally you must pass a parameter to the starting location to begin reading files, eg / or /mnt/ext_drive, etc. :

#!/bin/bash

# NAME: catall.sh (cat every single file)
# PATH: $HOME/askubuntu/
# DESC: Answer for: https://askubuntu.com/questions/1184103/trying-to-cp-a-directory-to-dev-null/1184105#1184105
# DATE: October 27, 2019.

[[ $(id -u) != 0 ]] && { echo "Must be called with sudo" >&2 ; exit 2 ; }
[[ $1 != /* ]] && { echo "Parameter 1 must be a path" >&2 ; exit 3 ; }


DirCnt=0    # Directory count
FileCnt=0   # File
LinkCnt=0   # Symbolic Link
BdevCnt=0   # Block Device
CdevCnt=0   # Character Device
PipeCnt=0   # Pipe
SockCnt=0   # Socket
ZeroCnt=0   # Zero sized files count
SkipCnt=0   # Files skipped

updatedb    # Update locate's database for files added today
StartSec=$SECONDS

while read File ; do
    if [[ ! -e "$File" ]] ; then : # File no longer exists, stale database
    elif [[ -d "$File" ]] ; then (( DirCnt++ ))
    elif [[ -h "$File" ]] ; then (( LinkCnt++ ))
    elif [[ -b "$File" ]] ; then (( BdevCnt++ ))
    elif [[ -b "$File" ]] ; then (( CdevCnt++ ))
    elif [[ -c "$File" ]] ; then (( CdevCnt++ ))
    elif [[ -p "$File" ]] ; then (( PipeCnt++ ))
    elif [[ -S "$File" ]] ; then (( SockCnt++ ))
    # elif [[ $File == /mnt/* ]] ; then (( SkipCnt ++ ))
    elif [[ -s "$File" ]] ; then
        cat "$File" 1>/dev/null
        (( FileCnt++ ))
    else
        (( ZeroCnt++ ))
    fi
    printf "Number of Directories: %'d Files: %'d \r" "$DirCnt" "$FileCnt"
done <<<"$(locate "$1")"

echo # Line feed to keep statistics visible when job ends
printf "Number of Sybolic Links: %'d\n"     "$LinkCnt"
printf "Number of Block Devices: %'d\n"     "$BdevCnt"
printf "Number of Character Devices: %'d\n" "$CdevCnt"
printf "Number of Pipes: %'d\n"             "$PipeCnt"
printf "Number of Sockets: %'d\n"           "$SockCnt"
printf "Number of Zero sized files: %'d\n"  "$ZeroCnt"
# printf "Number of Skipped files: %'d\n"     "$SkipCnt"
echo
EndSec=$SECONDS
TotalSec=$(( EndSec - StartSec ))
printf "Total Seconds: %'d\n"               "$TotalSec"

Two lines are commented out for skipping specific directories:

    # elif [[ $File == /mnt/* ]] ; then (( SkipCnt ++ ))
# printf "Number of Skipped files: %'d\n"     "$SkipCnt"

Remove the comment tag (#) and change /mnt/* to the directory you wish to skip to save time. In my case /mnt/ represents over a million files in Windows and other Ubuntu installations on different partitions.


Speed of cat is terrible

The final results were:

$ sudo ./catall.sh /

Number of Directories: 378,571 Files: 1,741,640 
Number of Sybolic Links: 92,011
Number of Block Devices: 25
Number of Character Devices: 55
Number of Pipes: 48
Number of Sockets: 138
Number of Zero sized files: 210,515

Total Seconds: 82,554

The process took 23 hours or almost a full day!

Much better speed can be achieved with dd even though a full partition is read including spaces where no files exist:

$ time sudo dd if=/dev/nvme0n1p4 of=/dev/null bs=100M
3719+1 records in
3719+1 records out
389969573888 bytes (390 GB, 363 GiB) copied, 249.867 s, 1.6 GB/s

real    4m9.896s
user    0m0.017s
sys     2m38.305s

Why use sudo with catall.sh script?

This example illustrates why sudo is required:

$ locate udev.log.1.gz
/mnt/clone/var/log/upstart/udev.log.1.gz
/var/log/upstart/udev.log.1.gz

$ cat /var/log/upstart/udev.log.1.gz | wc
cat: /var/log/upstart/udev.log.1.gz: Permission denied
      0       0       0

$ sudo cat /var/log/upstart/udev.log.1.gz | wc
      0       1      82
| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.