11

I'm trying to convert a Right Single Quotation Mark to an Apostrophe using tr.

tr "`echo -e '\xE2\x80\x99'`" "`echo -e '\x27'`" < a > b

given a UTF-8 encoded file called a which contains this example:

We’re not a different species
“All alone?” Jeth mentioned.

OS X uses the BSD tr and produces a nice result:

We're not a different species
“All alone?” Jeth mentioned.

Ubuntu uses the GNU tr and produces this nasty result:

We'''re not a different species
''<9C>All alone?''<9D> Jeth mentioned.

How can I accomplish this conversion in Ubuntu?

3

4 Answers 4

16

You could try some other tool, like sed:

$ sed "s/’/'/g" <a
We're not a different species
“All alone?” Jeth mentioned.

Or, since we are doing simple translation, use the y command for sed:

$ sed "y/’/'/" <a
We're not a different species
“All alone?” Jeth mentioned.

GNU tr doesn't work presumably because:

Currently tr fully supports only single-byte characters. Eventually it will support multibyte characters; when it does, the -C option will cause it to complement the set of characters, whereas -c will cause it to complement the set of values. This distinction will matter only when some values are not characters, and this is possible only in locales using multibyte encodings when the input contains encoding errors.

And is a multibyte character:

$ echo -n \' | wc -c
1
$ echo -n ’ | wc -c  
3
2
  • 1
    sed is much nicer for this kind of work.
    – Kaz Wolfe
    Nov 28, 2014 at 1:43
  • 2
    To explain the last part further: tr is replacing each of the three bytes separately with ', hence ''' as well as the broken sequences where it has replaced two of the three bytes in the similar characters and . It should instead understand the three bytes as together meaning one character, and replace that instead.
    – deltab
    Nov 28, 2014 at 4:37
9

If you also want to convert the double quotes, and perhaps other characters, you could use GNU iconv:

$ iconv -f utf-8 -t ascii//translit < a
We're not a different species
"All alone?" Jeth mentioned.

The //TRANSLIT suffix tells iconv that for characters outside the repertoire of the target encoding (here ASCII), it can substitute similar-looking characters or sequences automatically. Without the suffix, iconv will give up as soon as it finds an untranslatable character.

Note that //TRANSLIT seems to be a GNU extension: POSIX iconv doesn't support it.

3
  • +1. If you are converting a text from one character set (or encoding) to another, it can be sensible to use a tool designed for that purpose. Nov 28, 2014 at 11:59
  • @deltab your solution also replaces double quotation marks which OP don't want to replace them. Nov 28, 2014 at 12:27
  • @KasiyA Maybe they should.
    – gerrit
    Nov 28, 2014 at 16:43
3

You can use one of these awk solutions:

awk '{gsub(/\xE2\x80\x99/, "\x27");print}' file # with Hex ASCII code

awk '{gsub(/’/, "\x27");print}' file

awk '{gsub(/\342\200\231/, "\47");print}'  file # with Octal ASCII code

awk '{gsub(/’/, "\47");print}' file

Or

awk '{gsub(/’/, "'"'"'");print}' file
0

Use -s option of tr :

$ echo "We’re not a different species"|tr -s "’" "'"
We're not a different species

From man tr :

--truncate-set1
          first truncate SET1 to length of SET2
2
  • 1
    your solution also replaces double quotation marks which OP don't want to replace them Nov 28, 2014 at 12:25
  • Ah, indeed, thanks for pointing this out. I'll leave this answer for reference. Nov 28, 2014 at 12:43

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .