How can I find the line of corrupted characters (like �, ß, ø ...) in text files?

(A list of corrupted characters is here)

  • 3
    Not easily, but this is obviously caused by mismatched character encodings. Possibly the strings have been input as utf8 strings, but instead of being treated as utf8 strings, they've been treated as latin1 or some other single byte encoding. Figure out what encodings has been involved, and the question should be answerable. – geirha Aug 22 '12 at 18:18
  • 2
    There's no list at the link you gave. Could you edit your link to be more specific? By the way, I agree with Geirha. This is almost certainly related to character encoding. If you can figure out the original encoding and how it's now being interpreted, you should be able to convert the files easily using a tool such as iconv. – Scott Severance Aug 22 '12 at 23:00
  • @ScottSeverance: On the page I linked is a link to "Corrupted-original characters table (Excel file, 29Kb)" – Martin Thoma Aug 23 '12 at 7:21
  • if you have a text file, can you just use: iconv --from-code=UTF-8 (or --from-code=ASCII, etc)? The iconv utility will print "illegal input sequence at position {n}" on the line that has invalid data for the given input. You may also specific a target encoding; if the data can't be converted, you'll also receive a detailed error message. – michael Sep 19 '14 at 8:48

Though I haven't tested that particular specification inside the excel file I see the sql file mappings. You just asked to locate the bad characters, not fix them like the SQL function does. From stackoverflow to just see the line of non ascii characters:

grep --color='auto' -P -n "[\x80-\xFF]" file.xml

This will give you the line number, and will highlight non-ascii chars in red.

  • 1
    +1, I tested this solution, and it works for me :-) – sudodus Nov 14 '17 at 6:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.