7

Some text files I come across, have little squares with numbers in them (in place of certain characters). I am unable to copy and paste these in Ubuntu, but may search and replace in gedit each character individually (replacing for what I think is it's best match), obviously this is only feasible if there are only a few types of square.

An example of several of the squares

I'm lead to believe that these squares are displayed because I am missing certain fonts... My aim is to convert this into an ePub or PDF file.

My question is:

  • What type of coding is this? And why does this happen?
  • If it is missing fonts, can I install them and will this solve the problem (allow me to convert symbols to PDF e.g. using Calibre)?
  • Is there an application to convert my text file to a text file without these squares, instead replacing them with a similar character? For example, the symbol enter image description here is pretty much a y, so I would like this function to replace each instance of enter image description here with a y.

An example txt file is here and it originally looked like this (note inaccuracies followed OCR).

Note: I couldn't get either uni2ascii or iconv to work (though I may not have been using the correct [options]), so please check with the given file before posting a solution!

4
  • I actually don't see any stray characters in the sample file.
    – Amanda
    Jun 6, 2011 at 22:36
  • I just see: "ancient beau a temperate"
    – Amanda
    Jun 6, 2011 at 23:06
  • @Amanda do you have any additional language packs installed? / any idea why this could be?
    – hayd
    Jun 9, 2011 at 9:31
  • Having same issue, but I'm printing a webpage to PDF but my box-symbol has inside 'F022'. I was only able to find the symbol on the following website toward the middle under Java: fileformat.info/info/unicode/char/f022/index.htm The website indicates that it might have something to do with Java: string.toUpperCase() string.toLowerCase() Sorry, I don't know the answer, but maybe this might help us get closer to a solution....
    – user268860
    Apr 14, 2014 at 17:19

3 Answers 3

3

The boxes mean "glyph not found"; the characters in the box are hexidecimal representations of the codepoint, in unicode.

There are two possibilities: the character encoding is garbled, or the font you are using doesn't have a glyph for that character. This is a great overview character encoding if you really want to understand it: http://trochee.net/2011/05/character-encoding-tutorial/

Curiously, U+001F and U+001D are really just glorified line breaks. It seems odd that OCR would return those.

1
  • It's strange if they are "glorified line breaks"... why does gedit not simply display them as such! Indeed, why the tesseractOCR outputting them instead of a line break!
    – hayd
    Jun 9, 2011 at 9:35
1

The squares (as far as I can tell) always occur in places where special typesetting characters have been used. For example, typesetting ty as the letter t followed by the letter y in some fonts leaves extra, unwanted space between the two letters. For that reason, many fonts used for more advanced typesetting have extra characters for this, like the ty character that should read "...ancient beauty a temperate...". Since you don't have these extra characters (it's possible you can't even decode them, since they might not have an ascii/utf-8 code) you get squares.

I have no real idea on how to copy the actual text (and in this case get a t and a y as separate characters), but the people at TeX, LaTeX and friends might be able to help - they're not necessarily font experts, but they're all into typesetting...

1
0

That's not any encoding I recognize. My guess is that the missing symbols do not represent written characters, but rather indicate extra information about the OCR process.

Using a flexible interpretation of ASCII control codes, 0C might represent a page break and 0B could be a tab or other whitespace. 1D and 1F are supposed to be "delimiters to mark fields of data structures", but at a glance 1F could conceivably have been co-opted to mean unidentified:

$ hexdump -C -s 0xa0 myfile.txt | grep -C 1 " 1f "
00000250  6c 64 20 6f 66 20 61 6e  63 69 65 6e 74 20 62 65  |ld of ancient be|
00000260  61 75 1f 20 61 20 74 65  6d 70 65 72 61 74 65 2c  |au. a temperate,|
00000270  20 68 75 6d 69 64 20 72  65 67 69 6f 6e 20 77 68  | humid region wh|
00000280  6f 73 65 20 0a 6d 69 73  1f 20 75 6e 64 75 6c 61  |ose .mis. undula|
00000290  74 69 6e 67 20 68 69 6c  6c 73 20 68 61 64 20 62  |ting hills had b|
--
00000350  20 33 30 30 20 0a 73 70  65 63 69 65 73 20 6f 66  | 300 .species of|
00000360  20 74 72 65 65 73 20 67  72 65 1f 20 69 6e 63 6c  | trees gre. incl|
00000370  75 64 69 6e 67 20 6d 61  70 6c 65 73 2c 20 63 61  |uding maples, ca|
--
000006a0  65 20 61 62 6f 75 74 20  31 30 20 6b 69 6c 6f 6d  |e about 10 kilom|
000006b0  65 74 72 65 73 20 61 77  61 1f 20 62 65 79 6f 6e  |etres awa. beyon|
000006c0  64 20 61 20 70 61 73 73  20 0a 63 61 6c 6c 65 64  |d a pass .called|

In this sample, the byte 1F is being used degenerately in place of ty,, w,, and y,.

Another possibility is that the file was damaged during some past encoding conversion. Perhaps metadata specifying symbol fonts was discarded, or more meaningful, out-of-range characters were collapsed into ASCII. This would be consistent with the characters originally being rare ligatures.

In any case, the information required to programmatically translate it certainly not included in the file. Unless you can rerun the OCR, I think you're out of luck.

2
  • I'll have a go rerunning OCR myself (it may be more accurate since it was last run).
    – hayd
    Jun 7, 2011 at 10:52
  • I ran OCR again (using tesseract)and it was, for the most part an improvement, however there were still some squares throughout the text file.
    – hayd
    Jun 9, 2011 at 9:23

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .