7

The files I am trying to find/list are:

  • Any size (0 bytes accepted)
  • Consist only of ASCII NUL characters (0x00)
  • If there are any characters other than 0x00, the file shouldn't be listed.

The command I have now is:

grep -RLP '[^\x00]' .

Which works, but it also finds file which consists only of two bytes: 0xFF, 0xFE. Don't know why.

Is there any better command to find such files?

  • Note the default system encoding for Ubuntu is UTF-8, not ASCII. Though up to byte 0x7F, they're identical. – wjandrea Aug 17 '18 at 0:12
9

In short, what is happening here is that grep is trying to interpret your file as Unicode data. The sequence 0xFF, 0xFE is a Byte Order Marker for UTF-16.

(In my testing, even other sequences involving two 0xFF's or two 0xFE's etc. would still not match the '[^\x00]' regex, since even when trying to do UTF-8 these would be considered non-characters.)

Using a locale that doesn't use Unicode for character types should fix this, which you can accomplish by setting the LC_CTYPE environment variable. Use the C locale to force ASCII encoding (so no Unicode enabled):

LC_CTYPE=C grep -RLP '[^\x00]' .

UPDATE: As pointed out by @steeldriver, grep still acts on a line-by-line basis, so files containing NUL bytes and newlines will still match.

@DavidFoerster's solution using grep's -z does a good job of solving this problem, using the NUL bytes as separators does the trick.

Alternatively, I came up with a short Python 3 script (allzeroes.py) to check whether the file's contents are all zeroes:

#!/usr/bin/python3
import sys
assert len(sys.argv) == 2
with open(sys.argv[1], 'rb') as f:
    for block in iter(lambda: f.read(4096), b''):
        if any(block):
            sys.exit(1)

Which you can use in a find to locate all matches recursively:

$ find . -type f -exec allzeroes.py {} \; -print

I hope that helps.

| improve this answer | |
  • 3
    +1 although since grep is line-based, this will also output files that consist entirely of newlines - you may be able to work around that by specifying null-terminated mode using -z (although that will slurp any regular text files wholly into memory). Also I don't think -P is required here? – steeldriver Aug 17 '18 at 1:23
  • 1
    I am not happy that find will execute the python script for each file. This should be made all in Python script. Eventually even Perl. – pbies Dec 7 '19 at 20:25
  • @pbies True, but on the other hand as it stands this script can be used in different scenarios. It tries to do one thing only, in good old Unix fashion. That way, it's possible to compose it in ways that wouldn't be possible if it was only usable by traversing a directory tree. – filbranden Dec 7 '19 at 22:11
  • @filbranden we have one scenario here and one question. Answer should be fitting the question. – pbies Dec 8 '19 at 11:18
2

You can abuse grep’s alternative null-terminated line mode and thus search for files that contain only empty lines:

grep -L -z -e . ...

Replace ... with the file set that you want to scan (here: -R .).

Explanation

  • -z, --null-data – Treat the input as a set of lines, each terminated by a zero byte (the ASCII NUL character) instead of a newline.1
  • -e . – Use . as the search pattern, i. e. match any character.
  • -L, --files-without-match – Suppress normal output; instead print the name of each input file from which no output would normally have been printed. The scanning will stop on the first match.1

Test case

Set-up:

: > empty
truncate -s 100 zero
printf '%s\0' foo bar > foobar

Run test:

$ grep -L -z -e . empty zero foobar
empty
zero

1 From the grep(1) manual page.

| improve this answer | |
0

I'll provide another answer, which is script I am using. Runned from specific folder will recurse and list all the NUL files:

shopt -s globstar
for file in ./**
do
    [ -d "$file" ] || LC_CTYPE=C grep -qP '[^\x00]' "$file" || echo "$file"
done
| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.