I have written a Python3 appindicator which calls fortune and captures the output for display in the on-screen notification.

Some fortunes contain squares with a hexadecimal number when the corresponding glyph does not exist in the current font. Each square is the representation of the hexadecimal Unicode code point for the missing glyph.

I want to remove the hexadecimal text before I display to the user. I was hoping to find some Python API which would let me inspect text, character by character, to determine something like char.isValidCodePoint() or similar but I cannot find as such.

I found a possible solution that I wanted to investigate here but after installing fonttools via the terminal, my Python program could not import fonttools/fontTools.

Any ideas - either using the Python API or calling out to a terminal?

Update #1: I have since realised the fonttools sample code from the link above will not work for me as it is Python2. I suppose if fonttools could somehow be used, I could invoke a Python2 interpreter from my Python3 script.

Update #2: After lots of reading (see references below), I have since found fc-match but it cannot always uniquely identify the font in use. I obtain the current font in Python:

from gi.repository import Gio
fontName = Gio.Settings( "org.gnome.desktop.interface" ).get_string( "font-name" )

resulting in Ubuntu 11. Passing this result to pango-view along with the hexadecimal character, I get a list of fonts including Ubuntu. To my thinking if the glyph was NOT rendered by the font, the font should not appear in the result from pango-view!



This is a different approach from where you were going with this, but perhaps you could just use python's str.replace() or re.sub() methods to parse out the hexidecimal strings from your text body. i.e.:

If the hex is predictable:

originalText = "\xc3\xa5Test"
filteredText = originalText.replace("\xc3\xa5", "")

Or if you need to match any hex chars with a regular expression:

import re

originalText = "\xc3\xa5Test"
filteredText = re.sub(r'[^\x00-\x7f]', r'', originalText)

More good discussion of this strategy

| improve this answer | |
  • It's OK to give alternate options but I think you could improve your answer a lot by 1) adding short sample code 2) describing possible PROs and CONs from the original post proposed solution and yours. – lpanebr Nov 10 '17 at 18:51
  • 1
    I'm not attempting to critique the original solution so I don't know if a PRO/CON comparison will be helpful here. I have, however added code examples for my suggestions to the answer. – Christopher Hunter Nov 10 '17 at 20:57
  • @ChristopherHunter: I added a comment back in November 2017 that the text comes from fortune text files and the hexadecimal appears on rendering. That is incorrect! Rather, the hexadecimal is present in the fortune file. For example, in /usr/share/games/fortunes/computers the line "*** System shutdown message from root ***" has hexadecimal characters at the start of the line. Need to figure out how to remove those (which actually match a '?' and not remove legitimate '?'). – Bernmeister Sep 12 at 4:42

Unicode shaping engine

Use a Unicode shaping engine like Harfbuzz to detect missing glyph. Here is a working example:

from pyharfbuzz import shape
f = "/usr/local/lib/python3.6/site-packages/werkzeug/debug/shared/ubuntu.ttf"
t = "®"
s = shape(f, t)
t = "რ"
s = shape(f, t)



Here the output in IDLE3 while checking:

>>> t = "®"
>>> s = shape(f, t)
>>> s
[{'cluster': 0, 'glyph_name': 'registered', 'x_advance': 29.453125, 'y_advance': 0.0, 'x_offset': 0.0, 'y_offset': 0.0}]
>>> t = "რ"
>>> s = shape(f, t)
>>> s
[{'cluster': 0, 'glyph_name': '.notdef', 'x_advance': 36.0, 'y_advance': 0.0, 'x_offset': 0.0, 'y_offset': 0.0}]

Check for correct font path, i just picked the first one i saw in my current machine.


  • I'm quiet sure Gtk/Pango have similar function, Pango already switched to use Harfbuzz at low level. However, I don't have experience using such lib.
| improve this answer | |

Have come up with a solution...initially I thought the fortune text files did NOT contain the hexadecimal characters. Turns out that was incorrect. So once I realised this I came up with the solution below:

import codecs
fortune = <call the fortune program>
output = ""
for c in fortune:
    if codecs.encode( str.encode( c ), "hex" ) == b'07':

    output += c                   

print( output )
| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.