4

I am trying to find the size of the files on my hard drive in exact bytes, but whenever the size gets too big the number turns all weird (like 1.98329e+12). Can I stop it from doing this or convert this into exact bytes?

The command is:

ls -lR | grep -v '^d' | awk '{total += $5} END {print "Total:", total}'

Picture of exact bytes:

img

Picture of weird number:

img

  • The cut-off point before it stops showing exact bytes seems to be around 500gb
  • The command du -sb properly shows exact bytes no matter how big the directory is.
  • I have tried Ubuntu Gnome 15.10 64bit (Japanese and English) and Linux Mint 17.3 Cinnamon 64bit (Japanese)
  • My drives are ntfs so I tried formatting one as ext4 and copying my files over. The results are same as ntfs.
8
  • 5
    Use du command for size. ls is unfit for that purpose Apr 7, 2016 at 13:45
  • I would prefer ls because directory sizes vary while file sizes are constant
    – パンツ
    Apr 7, 2016 at 13:57
  • 1
    A lot of people prefer ls because it seems simple, but because its simple it has a lot of issues and generally is only good for listing files. Tasks such yours require different tools, and recursive behavior. Also, can you explain what do you mean by file and directory sizes being different and constant ? Unless you move files around on the drive, directory should remain the same too. Apr 7, 2016 at 14:03
  • 1
    You blame ls but clearly the problem is with the way awk displays the final result. The answer by @kos shows how to correct this.
    – alexis
    Apr 7, 2016 at 15:45
  • @alexis There's a lot more going on behind the scenes.AWK merely operates on the text. ls on the other hand is the one giving output to awk , and that data can be incorrect in a lot of ways. For example ls -l /dev/sda will not show you the disk usage of sda as many people think. ls should be used for only listing stuff, nothing more. OP's purpose is to find total disk size of all the files. Apr 7, 2016 at 17:30

4 Answers 4

5

The problem is that MAWK (the AWK variant installed on Ubuntu) by default prints integers bigger than 2147483647 (231-1) in scientific notation:

% awk -W version
mawk 1.3.3 Nov 1996, Copyright (C) Michael D. Brennan

compiled limits:
max NF             32767
sprintf buffer      2040
% printf '2147483647\n' | awk '{x += $1; print x}'
2147483647
% printf '2147483648\n' | awk '{x += $1; print x}'
2.14748e+09

You could use printf with a format specifer instead of print*:

printf '2147483648\n' | awk '{x += $1; printf "%.0f\n", x}'
% printf '2147483648\n' | awk '{x += $1; printf "%.0f\n", x}'
2147483648

In your case:

ls -lR | grep -v '^d' | awk '{total += $5} END {printf "Total:%.0f\n", total}'
ls -lR |
    grep -v '^d' |
    awk '
        {
            total += $5
        }
        END {
            printf "Total:%.0f\n", total
        }
    '

That will force AWK to print total in decimal notation instead of in scientific notation.

However, on another note, you should never parse ls.

A more sensitive way to do that would be using find + stat:

find . -type f -exec stat -c '%s' {} + | awk '{total += $1} END {printf "Total:%.0f\n", total}'
find . -type f -exec stat -c '%s' {} + |
    awk '
        {
            total += $1
        }
        END {
            printf "Total:%.0f\n", total
        }
    '

*%.0f is a trick to make printf print numbers bigger than 2147483647 (231-1), which when using %d as the format specifier would always print as 2147483647. The limit of %.0f is that one will start losing precision after 9007199254740992 (253), if that's ever a concern (thanks to Rotsor for the useful information).

9
  • 1
    Ok but i would use du with -b flag instead of stat Apr 7, 2016 at 14:05
  • @Serg Why exactly? In case of filenames containing newlines it's subject to the same pitfall as ls -l | grep -v '^d', and will print two lines instead of one. One could use du -0, but then one should also set AWK to read NUL-terminated strings. I don't see a reason to use du in place of stat in this case.
    – kos
    Apr 7, 2016 at 14:18
  • 1
    Aside form having -b flag to produce size in bytes as OP wants, there is -c flag, which produces grand total. So there's no need for awk that way. In addition, OP is not concerned with printing the file , so there's no need to be concerned about newlines Apr 7, 2016 at 14:48
  • I've posted that as an answer Apr 7, 2016 at 14:58
  • 1
    The first number you lose precision at is going to be 9007199254740993 (2^53 + 1) (8 PiB). This is because double-precision floating-point numbers have 52-bit mantissa, resulting in 53-bit effective precision.
    – Rotsor
    Apr 7, 2016 at 20:08
5

TL;DR: ls and awk are unnecessary for your purpose. Use du -cb or du -bs on the directory that you want to analyse.

Your purpose is to

  1. Find all files
  2. find their size (in bytes)
  3. produce grand total for all of them

All these actions can be performed by du.

$ du -bs $HOME 2>/dev/null                                                                 
76709521942 /home/xieerqi

It's worth noticing that du has two "modes" - it can either show how much file is in size OR how much actual disk space it takes up (the real , physical real-estate). Since you are interested in total size of all files, you want the apparent file size. -b flag gives exactly that ( -b is alias for --apparent-size --block-size=1 ).

Perhaps even more concise and appropriate solution would be to use du -bc directly on the directory you want. For instance, my home directory is about 76 GB in size

$ du -bc $HOME 2> /dev/null  | tail -1                    
76694582570 total

For some reason you worry about difference in folder size and file size.You said in the comments:

I would prefer ls because directory sizes vary while file sizes are constant

du is recursive, and sums up the file sizes. Also, a directory does have a static size of 4096 bytes ( 4k ) , but with du it will be included in the result of du -bs directory_name . Consider this:

$ du -b suse/openSUSE-Leap-42.1-DVD-x86_64.iso                                             
4648337408  suse/openSUSE-Leap-42.1-DVD-x86_64.iso

$ du -b suse/                                                                              
4648341504  suse/

$ bc <<< "4648337408+4096" 
4648341504

$ mkdir suse/another_dir  

$ du -b suse/another_dir                                                                   
4096    suse/another_dir

$ du -bs suse/                                                                             
4648345600  suse/
5
  • @kos -s produces total for each argument, so if arguments were directories, it would produce summary for all of them and output in for size dir_name , so last line would be whatever last argument was. -c actually produces the sum of sizes of all the arguments given and gives last line as grand total :) Apr 7, 2016 at 15:10
  • I meant a literal du -s ., not find . -type f -exec du -s {} \+ 2> /dev/null | awk 'END{print $0}' (and by the way since they want bytes I'll correct that to du -bs .).
    – kos
    Apr 7, 2016 at 15:26
  • @kos I've added that to my answer :) Apr 7, 2016 at 15:44
  • I was just re-reading and tested again, sorry, I didn't know -b implied --apparent-size. Removed that comment (and the imprecise first comment).
    – kos
    Apr 7, 2016 at 15:47
  • 1
    I always use du -cks which is useful, easy to remember, and funny.
    – MarkHu
    Apr 7, 2016 at 20:16
4

Under the hood, awk does all calculations using double-precision floating point numbers. By default it prints them using printf(3) format specifier %.6g, which means that if the number is more than six digits wide it will switch over to E-notation, which is what you saw. You can work around this by setting the variable OFMT:

ls -lR |
    awk 'BEGIN { OFMT = "%d" }  
         /^-/  { total += $5 } 
         END   { print "Total:", total }'

But there is an upper limit, beyond which it can't give you an exact number of bytes; it will start rounding off the low bits of the sum. 500 gigabytes = 500 * 1024 * 1024 * 1024 = 536870912000 ≈ 239. With the usual IEEE floating point, this is safely below that limit (which is roughly 252). However, it is large enough that I personally would feel better using a programming language that had proper "bignums" (unlimited-size integers). For instance, Python:

#! /usr/bin/python
import os
import sys

space = 0L  # L means "long" - not necessary in Python 3
for subdir, dirs, files in os.walk(sys.argv[1]):
    for f in files:
        space += os.lstat(os.path.join(subdir, f)).st_size

sys.stdout.write("Total: {:d}\n".format(space))

This is also completely immune to problems with files with unusual characters in their names. And it counts space consumed by hidden files.

This computes the number of bytes visible in each file, which is the same as what ls -l prints. If you want number of bytes actually occupied on disk instead (what du prints), replace .st_size with .st_blocks * 512. (Yes, the multiplier is always 512, even if st_blksize is a different number.)

5
  • You've mixed up the order. Read the documentation . Order should be dirpath, dirnames, filenames Apr 7, 2016 at 18:27
  • @Serg I just fixed that :)
    – zwol
    Apr 7, 2016 at 18:28
  • I've had a small issue with your code, so had to ask a question. See my post stackoverflow.com/q/36484470/3701431 Apr 7, 2016 at 18:43
  • 1
    +1 but on a side note the upper limit before the switch to a scientific notation is not exact, neither in GAWK nor in MAWK (the default AWK variant on Ubuntu, where the upper limit is higher, 2^32, i.e. the usual maximum size of an int, the maximum number representable being 2147483647). GAWK seems to have an even bigger upper limit, though I didn't test it nor I went through the documentation / source code (yet).
    – kos
    Apr 7, 2016 at 18:58
  • 1
    @Serg Gah, I should have thought of that - answer is now corrected. The right fix for the symlink problem you found is to use os.lstat instead of os.stat, because in this context, you do want to count the size of the symlinks themselves and you don't want to count the size of what they point to (if the pointee is inside the tree it should be counted only once; if it's outside it shouldn't be counted at all).
    – zwol
    Apr 8, 2016 at 20:34
3

What you see here is a way to display large numbers. For example:

1.23e+3 = 1.23*10^3 = 1230

As far as I know, you cannot turn this off, but as you wrote in your question, du does behave differently, so I would recommend to use this. Otherwise, you would have to convert the numbers.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .