You can print all non-ASCII lines of a file using my Python 3 script that I am hosting on GitHub here:
GitHub: ByteCommander/encoding-check
You can either clone or download the entire repository or simply save the file encoding-check
and make it executable using chmod +x encoding-check
.
Then you can run it like this, with the file to check as only argument:
./encoding-check FILENAME
if it's located in your current working directory, or...
/path/to/encoding-check FILENAME
if it's located in /path/to/
, or...
encoding-check FILENAME
if it's located in a directory that is part of the $PATH
environment variable, i.e. /usr/local/bin
or ~/bin
.
Without any optional arguments, it will print each line and its number where it found non-ASCII characters. Finally, there's a summary line that tells you how many lines the file had in total and how many of them contained non-ASCII characters.
This method is guaranteed to properly decode all ASCII characters and detect everything that is definitely not ASCII.
Here's an example run on a file containing the first 20 lines of your given install.en.txt
:
$ ./encoding-check install-first20.en.txt
9: Appendix��F, GNU General Public License.
14: (codename "���Xenial Xerus���"), for the 64-bit PC ("amd64") architecture. It also
18: ���������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������
--------------------------------------------------------------------------------
20 lines in 'install-first20.en.txt', thereof 3 lines with non-ASCII characters.
But the script has some additional arguments to tweak the checked encoding and the output format. View the help and try them:
$ encoding-check -h
usage: encoding-check [-h] [-e ENCODING] [-s | -c | -l] [-m] [-w] [-n] [-f N]
[-t]
FILE [FILE ...]
Show all lines of a FILE containing characters that don't match the selected
ENCODING.
positional arguments:
FILE the file to be examined
optional arguments:
-h, --help show this help message and exit
-e ENCODING, --encoding ENCODING
file encoding to test (default 'ascii')
-s, --summary only print the summary
-c, --count only print the detected line count
-l, --lines only print the detected lines
-m, --only-matching hide files without matching lines from output
-w, --no-warnings hide warnings from output
-n, --no-numbers do not show line numbers in output
-f N, --fit-width N trim lines to N characters, or terminal width if N=0;
non-printable characters like tabs will be removed
-t, --title print title line above each file
As --encoding
, every codec that Python 3 knows is valid. Just try one, in the worst case you get a little error message...
.decode("utf8", "strict")
method. I could not find any line that wouldn't decode this way.file
being your example file,file1
being a file containing onlyàèìòù
followed by a newline). Is this GNUgrep
?curl
piped togrep
, to get the file directly from source?