11

I'm trying to convert a Right Single Quotation Mark to an Apostrophe using tr.

tr "`echo -e '\xE2\x80\x99'`" "`echo -e '\x27'`" < a > b

given a UTF-8 encoded file called a which contains this example:

We’re not a different species
“All alone?” Jeth mentioned.

OS X uses the BSD tr and produces a nice result:

We're not a different species
“All alone?” Jeth mentioned.

Ubuntu uses the GNU tr and produces this nasty result:

We'''re not a different species
''<9C>All alone?''<9D> Jeth mentioned.

How can I accomplish this conversion in Ubuntu?

16

You could try some other tool, like sed:

$ sed "s/’/'/g" <a
We're not a different species
“All alone?” Jeth mentioned.

Or, since we are doing simple translation, use the y command for sed:

$ sed "y/’/'/" <a
We're not a different species
“All alone?” Jeth mentioned.

GNU tr doesn't work presumably because:

Currently tr fully supports only single-byte characters. Eventually it will support multibyte characters; when it does, the -C option will cause it to complement the set of characters, whereas -c will cause it to complement the set of values. This distinction will matter only when some values are not characters, and this is possible only in locales using multibyte encodings when the input contains encoding errors.

And is a multibyte character:

$ echo -n \' | wc -c
1
$ echo -n ’ | wc -c  
3
  • 1
    sed is much nicer for this kind of work. – Kaz Wolfe Nov 28 '14 at 1:43
  • 2
    To explain the last part further: tr is replacing each of the three bytes separately with ', hence ''' as well as the broken sequences where it has replaced two of the three bytes in the similar characters and . It should instead understand the three bytes as together meaning one character, and replace that instead. – deltab Nov 28 '14 at 4:37
  • For good understand is a multibyte character also we can use tr -c '[:print:][:cntrl:]' '-' command to replace every non-printing character, other than valid control characters, with a -. And you will see single translate to 3 bytes of characters like ---. good point for multi-byte character. – αғsнιη Nov 28 '14 at 9:01
9

If you also want to convert the double quotes, and perhaps other characters, you could use GNU iconv:

$ iconv -f utf-8 -t ascii//translit < a
We're not a different species
"All alone?" Jeth mentioned.

The //TRANSLIT suffix tells iconv that for characters outside the repertoire of the target encoding (here ASCII), it can substitute similar-looking characters or sequences automatically. Without the suffix, iconv will give up as soon as it finds an untranslatable character.

Note that //TRANSLIT seems to be a GNU extension: POSIX iconv doesn't support it.

  • +1. If you are converting a text from one character set (or encoding) to another, it can be sensible to use a tool designed for that purpose. – RedGrittyBrick Nov 28 '14 at 11:59
  • @deltab your solution also replaces double quotation marks which OP don't want to replace them. – αғsнιη Nov 28 '14 at 12:27
  • @KasiyA Maybe they should. – gerrit Nov 28 '14 at 16:43
3

You can use one of these awk solutions:

awk '{gsub(/\xE2\x80\x99/, "\x27");print}' file # with Hex ASCII code

awk '{gsub(/’/, "\x27");print}' file

awk '{gsub(/\342\200\231/, "\47");print}'  file # with Octal ASCII code

awk '{gsub(/’/, "\47");print}' file

Or

awk '{gsub(/’/, "'"'"'");print}' file
0

Use -s option of tr :

$ echo "We’re not a different species"|tr -s "’" "'"
We're not a different species

From man tr :

--truncate-set1
          first truncate SET1 to length of SET2
  • 1
    your solution also replaces double quotation marks which OP don't want to replace them – αғsнιη Nov 28 '14 at 12:25
  • Ah, indeed, thanks for pointing this out. I'll leave this answer for reference. – Skippy le Grand Gourou Nov 28 '14 at 12:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.