4

I am trying to find the size of the files on my hard drive in exact bytes, but whenever the size gets too big the number turns all weird (like 1.98329e+12). Can I stop it from doing this or convert this into exact bytes?

The command is:

ls -lR | grep -v '^d' | awk '{total += $5} END {print "Total:", total}'

Picture of exact bytes:

img

Picture of weird number:

img

  • The cut-off point before it stops showing exact bytes seems to be around 500gb
  • The command du -sb properly shows exact bytes no matter how big the directory is.
  • I have tried Ubuntu Gnome 15.10 64bit (Japanese and English) and Linux Mint 17.3 Cinnamon 64bit (Japanese)
  • My drives are ntfs so I tried formatting one as ext4 and copying my files over. The results are same as ntfs.
8
  • 5
    Use du command for size. ls is unfit for that purpose – Sergiy Kolodyazhnyy Apr 7 '16 at 13:45
  • I would prefer ls because directory sizes vary while file sizes are constant – パンツ Apr 7 '16 at 13:57
  • 1
    A lot of people prefer ls because it seems simple, but because its simple it has a lot of issues and generally is only good for listing files. Tasks such yours require different tools, and recursive behavior. Also, can you explain what do you mean by file and directory sizes being different and constant ? Unless you move files around on the drive, directory should remain the same too. – Sergiy Kolodyazhnyy Apr 7 '16 at 14:03
  • 1
    You blame ls but clearly the problem is with the way awk displays the final result. The answer by @kos shows how to correct this. – alexis Apr 7 '16 at 15:45
  • @alexis There's a lot more going on behind the scenes.AWK merely operates on the text. ls on the other hand is the one giving output to awk , and that data can be incorrect in a lot of ways. For example ls -l /dev/sda will not show you the disk usage of sda as many people think. ls should be used for only listing stuff, nothing more. OP's purpose is to find total disk size of all the files. – Sergiy Kolodyazhnyy Apr 7 '16 at 17:30
5

The problem is that MAWK (the AWK variant installed on Ubuntu) by default prints integers bigger than 2147483647 (231-1) in scientific notation:

% awk -W version
mawk 1.3.3 Nov 1996, Copyright (C) Michael D. Brennan

compiled limits:
max NF             32767
sprintf buffer      2040
% printf '2147483647\n' | awk '{x += $1; print x}'
2147483647
% printf '2147483648\n' | awk '{x += $1; print x}'
2.14748e+09

You could use printf with a format specifer instead of print*:

printf '2147483648\n' | awk '{x += $1; printf "%.0f\n", x}'
% printf '2147483648\n' | awk '{x += $1; printf "%.0f\n", x}'
2147483648

In your case:

ls -lR | grep -v '^d' | awk '{total += $5} END {printf "Total:%.0f\n", total}'
ls -lR |
    grep -v '^d' |
    awk '
        {
            total += $5
        }
        END {
            printf "Total:%.0f\n", total
        }
    '

That will force AWK to print total in decimal notation instead of in scientific notation.

However, on another note, you should never parse ls.

A more sensitive way to do that would be using find + stat:

find . -type f -exec stat -c '%s' {} + | awk '{total += $1} END {printf "Total:%.0f\n", total}'
find . -type f -exec stat -c '%s' {} + |
    awk '
        {
            total += $1
        }
        END {
            printf "Total:%.0f\n", total
        }
    '

*%.0f is a trick to make printf print numbers bigger than 2147483647 (231-1), which when using %d as the format specifier would always print as 2147483647. The limit of %.0f is that one will start losing precision after 9007199254740992 (253), if that's ever a concern (thanks to Rotsor for the useful information).

9
  • 1
    Ok but i would use du with -b flag instead of stat – Sergiy Kolodyazhnyy Apr 7 '16 at 14:05
  • @Serg Why exactly? In case of filenames containing newlines it's subject to the same pitfall as ls -l | grep -v '^d', and will print two lines instead of one. One could use du -0, but then one should also set AWK to read NUL-terminated strings. I don't see a reason to use du in place of stat in this case. – kos Apr 7 '16 at 14:18
  • 1
    Aside form having -b flag to produce size in bytes as OP wants, there is -c flag, which produces grand total. So there's no need for awk that way. In addition, OP is not concerned with printing the file , so there's no need to be concerned about newlines – Sergiy Kolodyazhnyy Apr 7 '16 at 14:48
  • I've posted that as an answer – Sergiy Kolodyazhnyy Apr 7 '16 at 14:58
  • 1
    The first number you lose precision at is going to be 9007199254740993 (2^53 + 1) (8 PiB). This is because double-precision floating-point numbers have 52-bit mantissa, resulting in 53-bit effective precision. – Rotsor Apr 7 '16 at 20:08
5

TL;DR: ls and awk are unnecessary for your purpose. Use du -cb or du -bs on the directory that you want to analyse.

Your purpose is to

  1. Find all files
  2. find their size (in bytes)
  3. produce grand total for all of them

All these actions can be performed by du.

$ du -bs $HOME 2>/dev/null                                                                 
76709521942 /home/xieerqi

It's worth noticing that du has two "modes" - it can either show how much file is in size OR how much actual disk space it takes up (the real , physical real-estate). Since you are interested in total size of all files, you want the apparent file size. -b flag gives exactly that ( -b is alias for --apparent-size --block-size=1 ).

Perhaps even more concise and appropriate solution would be to use du -bc directly on the directory you want. For instance, my home directory is about 76 GB in size

$ du -bc $HOME 2> /dev/null  | tail -1                    
76694582570 total

For some reason you worry about difference in folder size and file size.You said in the comments:

I would prefer ls because directory sizes vary while file sizes are constant

du is recursive, and sums up the file sizes. Also, a directory does have a static size of 4096 bytes ( 4k ) , but with du it will be included in the result of du -bs directory_name . Consider this:

$ du -b suse/openSUSE-Leap-42.1-DVD-x86_64.iso                                             
4648337408  suse/openSUSE-Leap-42.1-DVD-x86_64.iso

$ du -b suse/                                                                              
4648341504  suse/

$ bc <<< "4648337408+4096" 
4648341504

$ mkdir suse/another_dir  

$ du -b suse/another_dir                                                                   
4096    suse/another_dir

$ du -bs suse/                                                                             
4648345600  suse/
5
  • @kos -s produces total for each argument, so if arguments were directories, it would produce summary for all of them and output in for size dir_name , so last line would be whatever last argument was. -c actually produces the sum of sizes of all the arguments given and gives last line as grand total :) – Sergiy Kolodyazhnyy Apr 7 '16 at 15:10
  • I meant a literal du -s ., not find . -type f -exec du -s {} \+ 2> /dev/null | awk 'END{print $0}' (and by the way since they want bytes I'll correct that to du -bs .). – kos Apr 7 '16 at 15:26
  • @kos I've added that to my answer :) – Sergiy Kolodyazhnyy Apr 7 '16 at 15:44
  • I was just re-reading and tested again, sorry, I didn't know -b implied --apparent-size. Removed that comment (and the imprecise first comment). – kos Apr 7 '16 at 15:47
  • 1
    I always use du -cks which is useful, easy to remember, and funny. – MarkHu Apr 7 '16 at 20:16
4

Under the hood, awk does all calculations using double-precision floating point numbers. By default it prints them using printf(3) format specifier %.6g, which means that if the number is more than six digits wide it will switch over to E-notation, which is what you saw. You can work around this by setting the variable OFMT:

ls -lR |
    awk 'BEGIN { OFMT = "%d" }  
         /^-/  { total += $5 } 
         END   { print "Total:", total }'

But there is an upper limit, beyond which it can't give you an exact number of bytes; it will start rounding off the low bits of the sum. 500 gigabytes = 500 * 1024 * 1024 * 1024 = 536870912000 ≈ 239. With the usual IEEE floating point, this is safely below that limit (which is roughly 252). However, it is large enough that I personally would feel better using a programming language that had proper "bignums" (unlimited-size integers). For instance, Python:

#! /usr/bin/python
import os
import sys

space = 0L  # L means "long" - not necessary in Python 3
for subdir, dirs, files in os.walk(sys.argv[1]):
    for f in files:
        space += os.lstat(os.path.join(subdir, f)).st_size

sys.stdout.write("Total: {:d}\n".format(space))

This is also completely immune to problems with files with unusual characters in their names. And it counts space consumed by hidden files.

This computes the number of bytes visible in each file, which is the same as what ls -l prints. If you want number of bytes actually occupied on disk instead (what du prints), replace .st_size with .st_blocks * 512. (Yes, the multiplier is always 512, even if st_blksize is a different number.)

5
  • You've mixed up the order. Read the documentation . Order should be dirpath, dirnames, filenames – Sergiy Kolodyazhnyy Apr 7 '16 at 18:27
  • @Serg I just fixed that :) – zwol Apr 7 '16 at 18:28
  • I've had a small issue with your code, so had to ask a question. See my post stackoverflow.com/q/36484470/3701431 – Sergiy Kolodyazhnyy Apr 7 '16 at 18:43
  • 1
    +1 but on a side note the upper limit before the switch to a scientific notation is not exact, neither in GAWK nor in MAWK (the default AWK variant on Ubuntu, where the upper limit is higher, 2^32, i.e. the usual maximum size of an int, the maximum number representable being 2147483647). GAWK seems to have an even bigger upper limit, though I didn't test it nor I went through the documentation / source code (yet). – kos Apr 7 '16 at 18:58
  • 1
    @Serg Gah, I should have thought of that - answer is now corrected. The right fix for the symlink problem you found is to use os.lstat instead of os.stat, because in this context, you do want to count the size of the symlinks themselves and you don't want to count the size of what they point to (if the pointee is inside the tree it should be counted only once; if it's outside it shouldn't be counted at all). – zwol Apr 8 '16 at 20:34
3

What you see here is a way to display large numbers. For example:

1.23e+3 = 1.23*10^3 = 1230

As far as I know, you cannot turn this off, but as you wrote in your question, du does behave differently, so I would recommend to use this. Otherwise, you would have to convert the numbers.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.