8

The files I am trying to find/list are:

  • Any size (0 bytes accepted)
  • Consist only of ASCII NUL characters (0x00)
  • If there are any characters other than 0x00, the file shouldn't be listed.

The command I have now is:

grep -RLP '[^\x00]' .

Which works, but it also finds file which consists only of two bytes: 0xFF, 0xFE. Don't know why.

Is there any better command to find such files?

1
  • Note the default system encoding for Ubuntu is UTF-8, not ASCII. Though up to byte 0x7F, they're identical.
    – wjandrea
    Aug 17, 2018 at 0:12

4 Answers 4

9

In short, what is happening here is that grep is trying to interpret your file as Unicode data. The sequence 0xFF, 0xFE is a Byte Order Marker for UTF-16.

(In my testing, even other sequences involving two 0xFF's or two 0xFE's etc. would still not match the '[^\x00]' regex, since even when trying to do UTF-8 these would be considered non-characters.)

Using a locale that doesn't use Unicode for character types should fix this, which you can accomplish by setting the LC_CTYPE environment variable. Use the C locale to force ASCII encoding (so no Unicode enabled):

LC_CTYPE=C grep -RLP '[^\x00]' .

UPDATE: As pointed out by @steeldriver, grep still acts on a line-by-line basis, so files containing NUL bytes and newlines will still match.

@DavidFoerster's solution using grep's -z does a good job of solving this problem, using the NUL bytes as separators does the trick.

Alternatively, I came up with a short Python 3 script (allzeroes.py) to check whether the file's contents are all zeroes:

#!/usr/bin/python3
import sys
assert len(sys.argv) == 2
with open(sys.argv[1], 'rb') as f:
    for block in iter(lambda: f.read(4096), b''):
        if any(block):
            sys.exit(1)

Which you can use in a find to locate all matches recursively:

$ find . -type f -exec allzeroes.py {} \; -print

I hope that helps.

4
  • 3
    +1 although since grep is line-based, this will also output files that consist entirely of newlines - you may be able to work around that by specifying null-terminated mode using -z (although that will slurp any regular text files wholly into memory). Also I don't think -P is required here? Aug 17, 2018 at 1:23
  • 1
    I am not happy that find will execute the python script for each file. This should be made all in Python script. Eventually even Perl.
    – pbies
    Dec 7, 2019 at 20:25
  • @pbies True, but on the other hand as it stands this script can be used in different scenarios. It tries to do one thing only, in good old Unix fashion. That way, it's possible to compose it in ways that wouldn't be possible if it was only usable by traversing a directory tree.
    – filbranden
    Dec 7, 2019 at 22:11
  • @filbranden we have one scenario here and one question. Answer should be fitting the question.
    – pbies
    Dec 8, 2019 at 11:18
2

You can abuse grep’s alternative null-terminated line mode and thus search for files that contain only empty lines:

grep -L -z -e . ...

Replace ... with the file set that you want to scan (here: -R .).

Explanation

  • -z, --null-data – Treat the input as a set of lines, each terminated by a zero byte (the ASCII NUL character) instead of a newline.1
  • -e . – Use . as the search pattern, i. e. match any character.
  • -L, --files-without-match – Suppress normal output; instead print the name of each input file from which no output would normally have been printed. The scanning will stop on the first match.1

Test case

Set-up:

: > empty
truncate -s 100 zero
printf '%s\0' foo bar > foobar

Run test:

$ grep -L -z -e . empty zero foobar
empty
zero

1 From the grep(1) manual page.

0

I'll provide another answer, which is script I am using. Runned from specific folder will recurse and list all the NUL files:

shopt -s globstar
for file in ./**
do
    [ -d "$file" ] || LC_CTYPE=C grep -qP '[^\x00]' "$file" || echo "$file"
done
-1

You can use this PHP code for find all files with NULL contents.

<?php
$pattern = '';
$directory = new RecursiveDirectoryIterator("./");
$iterator = new RecursiveIteratorIterator($directory);
if ($pattern) {
    $regex = new RegexIterator($iterator, $pattern);
} else {
    $regex = $iterator;
}
foreach($regex as $file) {
    if (is_dir($file)) continue;
    $contens = file_get_contents($file);
    $contens = trim($contens);
    if (strlen($contens) == 0) {
        echo "$file\n";
    }
}
3
  • Getting all files into memory is a bad idea. Think what would happen if you have 100 GB file and you load it into memory. -1
    – pbies
    Nov 3, 2020 at 9:21
  • @pbies Each time the loop is executed, the variable "$contens" is override with new contents and does not increase the memory used. But I agree this solution is not speedly. Nov 4, 2020 at 7:05
  • Memory usage will be based on file's length. It is not advised to load whole file to memory when you need only to check few first bytes. Also PHP is slow.
    – pbies
    Nov 4, 2020 at 12:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.