Take the 2-minute tour ×
Ask Ubuntu is a question and answer site for Ubuntu users and developers. It's 100% free, no registration required.

The 'printf' I refer to is the standard-issue "program" (not the built-in): /usr/bin/printf

I was testing printf out as a viable method of convert a Unicode Codepoint Hex-literal into its Unicoder character representation,

I was looking good, and seemed flawless..(btw. the built-in printf can't do this at all (I think)...

I then thought to test it at the lower extreme end of the code-spectrum, and it failed with an avalanche of errors.. All in the ASCII range (= 7 bits)

The strangest thing was that 3 value printed normally; they are:

  • $ \u0024
  • @ \u0040
  • ` \u0060

I'd like to know what is going on here. The ASCII character-set is most definitely part of the Unicode Code-point sequence....

I am puzzled, and still without a good way to bash script this particular converion.. Suggestions are welcome.

To be entertained by that same avalanche of errors, paste the following code into a terminal...

# Here is one of the error messages
# /usr/bin/printf: invalid universal character name \u0041
#  ...for them all, run the following script
( 
for nib1 in {0..9} {A..F}; do 
  for nib0 in {0..9} {A..F}; do
   [[ $nib1 < A ]] && nl="\n" || nl=" "
   $(type -P printf) "\u00$nib1$nib0$nl"
  done
done 
echo 
)
share|improve this question
add comment

3 Answers 3

up vote 1 down vote accepted

The printf command has reasons for not accepting charicters in this range. If you look at the sounce code for for printf, you will see this comment:

A universal character name shall not specify a character short identifier in the range 00000000 through 00000020, 0000007F through 0000009F, or 0000D800 through 0000DFFF inclusive. A universal character name shall not designate a character in the required character set.

You might be able to recompile without that check, but it looks very deliberate to me. Try instead to use the command without the \u, for instance:

( 
for nib1 in {0..9} {A..F}; do 
  for nib0 in {0..9} {A..F}; do
   $(type -P printf) "\00$nib1$nib0"
  done
done 
echo 
)
share|improve this answer
    
Martin, Thanks for finding that info. Yes, they state it to be deliberate, but not why. The mystery is somewhat diffused, and yet somehow more intriguing.. (Re. dropping the '\u'.. it defeats my intent; it simply prints my hex-literal 'FF' back to me with '00' prepended; '00FF').. so, although you have uncovered a very relevant fact, I'm now up in the air about the 'why' behind their 'shall not'... What is so important about this issue that they specifally code around it?... (and it leaves me still looking for a 'light' way to convert a hex-literal into its Unicode CodePoint character –  Peter.O Jan 10 '11 at 3:48
    
Well you could just modify the code and recompile the package. But then again I bet those chars were blocked simply because they're not visible. –  Martin Owens -doctormo- Jan 10 '11 at 4:26
    
It blocks the entire Basic Latin Alphabet. not just control chars.. Is the source C++, because the only thing I've found on it is a document which mentions this issue is in relation to the C++ standards: open-std.org/jtc1/sc22/wg21/docs/papers/2007/n2170.html .. Maybe its because of some obscure double back-twist security related issue.. At least, I now know that I can forget about this printf being a simple one-stop shop for printing the full range of unicode chars from a hex literal (in a bash script)... but there must be something out there which fits the bill :) –  Peter.O Jan 10 '11 at 4:49
    
Ask the question, I'll answer it and get points! huzzar. –  Martin Owens -doctormo- Jan 10 '11 at 5:17
add comment

The three working characters are the three printable ASCII characters that are not in the C basic character set. The reason why these characters are forbidden in C is that it would be hard on compilers: they would need to perform the \u interpolation before lexical analysis, which I think would break in a few corner cases, and would be impractical in many compilers anyway (because characters outside the basic set are only required to be allowed in a few places).

Having the same forbidden characters doesn't make sense in a shell utility. I suspect this is a bug, and $, @ and ` should not work either. The reason not to support them is again to make it easier to parse strings. For example, if you want to determine that there is no special character in a string you're going to put into a database query, you can check that the string doesn't contain ', and not worry about it containing \u002a.

Consider using recode Install recode as suggested in the GNU coreutils manual, or (more portable in practice) Perl or python.

share|improve this answer
    
Gilles, thanks for the explanation... So it seems that some "body" (the C standards group, perhaps) has tried to inhibit coders from using these blocked "u.."encodings in the source-code itself.. Fine! .. but I don't see how that, in any way, is related to runtime use of printf, which has got nothing to do with the compiler, and only ever to do with user of a running app... I'm befuddled by this.. but it doesn't matter, because I now know roughly what is going on... and I've discovered some interesting tools along the way; eg. recode; which has a 0x1234 "surface", but no "U1234" –  Peter.O Jan 11 '11 at 4:27
add comment
(for nib1 in {0..9} {A..F}; do 
  for nib0 in {0..9} {A..F}; do
   $(type -P printf) "\x$nib1$nib0"
  done
done 
echo )

prints (hand format adjusted)

            


                    
   ! " # $ % & ' ( ) * + , - . / 0 1 2 3 4 5 6 7 8 9 : ; < = > ?
 @ A B C D E F G H I J K L M N O P Q R S T U V W X Y Z [ \ ] ^ _ 
 ` a b c d e f g h i j k l m n o p q r s t u v w x y z { | } ~  
 � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � �
 � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � �
 � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � �
 � � � � � � � � � � � � � � � � � � � � � � � � � � � � � �

Note that some "character" control codes "worked" ie. HT, VT, LF. etc.

share|improve this answer
    
The \x notation only caters for the range 00-FF (ie. only for a single byte). It has "guaranteed" equivalence to Unicode Codepoint values only in the ASCII range 00-7F. I was specifically interested in the Unicode \u notation which caters for Unicode Codepoints 0000-FFFF. –  Peter.O Mar 13 '13 at 18:55
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.