Custom Keyboard Variant (for Transliteration of Arabic) not working

Question

I tried setting up a custom keyboard layout in Ubuntu 18.04.1 LTS for being able to write Arabic transliteration for scientific purposes (see: https://en.wikipedia.org/wiki/Romanization_of_Arabic ; https://en.wikipedia.org/wiki/DIN_31635 ) Therefore I need to be able to type special characters.

I more or less followed these instructions:

• https://sites.lsa.umich.edu/kchalipa/?p=61
• https://aboutfoto.wordpress.com/2017/04/23/custom-keyboard-xkb/
• What are the steps needed to create new keyboard layout on ubuntu?

What I did:

1. As I speak German I opened "de" in /usr/share/X11/xkb/symbols/ with gedit and root privileges.

2. In this file I inserted the following at the end and saved (made a backup of the original before):

   // Deutsch mit orientalistischer Umschrift
   partial alphanumeric_keys
   xkb_symbols "german_orient" {
   
       include "de(basic)"
   
       name[Group1]="Deutsch (Orientalistische Umschrift)";
   
       key <BKSL> { [numbersign, apostrophe, U+02BE,   dead_breve ]    };
       key <AE12>  { [dead_acute, dead_grave, U+02BF,  dead_ogonek ]   };
       key <AC11>  { [adiaeresis, Adiaeresis, U+0304, dead_caron ] };
       key <AB10>  { [minus, underscore, U+0331, emdash ] };
       key <AC10>  { [odiaeresis, Odiaeresis, U+0307, dead_belowdot ] };
       key <AB09>  { [period, colon, U+0323, division ] };
       key <AB08>  { [comma, semicolon, U+030C, multiply   ] };
       key <AC06>  { [h, H, U+032E, NoSymbol ] };
   };
   

   In comparison to the standard keyboard layout in German for every key listed here the third entry (= pressing the key together with AltGr) was changed according to my needs. BTW I got the Unicode-Codes by looking up the special characters on the internet.

3. Then I opened evdev.xml in /usr/share/X11/xkb/rules/ in the same way

4. In this file I searched for the german (de) layout entry and inserted at its end (meaning before </layout>) the following and saved the file:

   <variant>
    <configItem>
    <name>german_orient</name>
    <description>Deutsch (Orientalistische Umschrift)</description>
    </configItem>
   </variant>
   

5. Then I typed the following command in the terminal and pressed enter sudo dpkg-reconfigure xkb-data because in many tutorials on the topic it is advised to do so

6. Then I did a reboot.

7. Then I added the new keyboard layout "Deutsch (Orientalistische Umschrift)" (meaning: German (orientalistic transliteration) in Ubuntu Preferences

The Result was not as I had hoped for:

Not only was I not able to write the special characters with the keys I had assigned. I guess I somehow "broke" the "de" keyboard layout because no matter if I activated the standard German layout or the "Deutsch (Orientalistische Umschrift)"-layout via super+space the system seemed to have fallen back to the default English keyboard layout. So even if one of the German layouts was activated the letters I wrote were according to the English layout (for example the letters "z" and "y" were switched).

So, in the end, I undid my changes using the backups of both files I had made and now everything is back to normal.

But as I need to be able to write Arabic transliteration fast for professional purposes and as there are - at least to my knowledge - no keyboard-layouts for this purpose available for download (as the number of scholars in this field using Linux is quite small) I need the help of someone here to tell me what to change in order to make my custom keyboard layout work.

I'm still a noob in Linux, so sorry if I missed something obvious. Anyway, thanks in advance for your help!

Update: Found a solution myself

I found a solution myself and I'm gonna post it here since someone else might profit from it

Install KeyboardLayoutEditor and edit the keyboard layout-file

1. I Downloaded and installed KeyboardLayoutEditor from https://github.com/simos/keyboardlayouteditor

   • When I tried to run it, I noticed that some packages were missing which I had to install first and which were not listed among the requirements.

2. Run KeyboardLayoutEditor and load a suitable language layout-file (de in my case)

   • Then choose the variant you want to edit (in my case "basic" since I figured I might have made some error while adding my own variant)

3. Then Click on the "Start Charakter Map" Button which will load Gucharmap (I had to install it first via Ubuntu Software Center for this to work)

4. Now choose the character you want to add to a certain key (you can also search for signs in Gucharmap by entering Unicode-Codes)

5. And drag and drop it to one of the 4 positions of a suitable key in KeyboardLayoutEditor (see below for the final code)

   • BTW I think the application made some mistakes here. Because with dragging some characters to a certain key, it didn't write the correct unicode-code into the de-file but some name of the corresponding deadkey character. (For example it wrote "dead_belowdot" instead of U0323 allthough I chose the latter character in Gucharmap and dragged it over. I suppose this is a bug.
   • The result was
     • that with these deadkey special characters I had to type them FIRST and THEN the letter I wanted them to appear over or under.
     • the other inserted characters with the correct unicode-code could be typed by FIRST typing the letter and THEN the special character I wanted to add above or below. This is the behaviour I am used to, so I tried to fix the deadkey versions:
   • My solution was to edit the de-file by hand replacing all the wrongly inserted deadkey-versions with the correct unicode-code for the combined character-version. Now I can type a letter and then add the special character.

6. When you inserted all characters you need you can save the file

   • In my case KeyboardLayoutEditor didn't just save the changes to the "basic" variant inside the "de-file" but overwrote the whole "de-file" thereby deleting all other variants
   • Allthough I think I don't need the other variants I didn't like that. So I copied the code KeyboardLayoutEditor wrote in the de-File to my de_backup replacing the "basic" section there and then saved de_backup as "de" replacing the old "de".
   • the evdev.xml doesn't have to be edited this time because an existing variant was edited so no new variant/layout has to be added to the file.

7. Then I used the sudo dpkg-reconfigure xkb-data command in the console

8. Finally I did a reboot

Result: Everythin works like a charm now! :)

Some thoughts

I don't know why it worked this time by using KeyboardLayoutEditor.

1. My first guess is that it worked because I didn't try to add a new variant to the keyboard-Layout-file (de) but edited an existing one thereby avoiding some possible mistakes with editing the de-file and the evdev-xml

2. My second guess is that it worked because KeyboardLayout editor wrote the Unicode-Codes different than I had when first manually editing the de-file.

   • For example the unicode-code I inserted looked like this: U+02BE
   • The unicode-code written by the application looks like this: U02BE
   • Perhaps by leaving out the "+" it worked? As I am a noob I don't know.
   • Some knowledgable people might see a a differend cause in the code below.

The Code which worked which I inserted in the keyboard layout-file (de in my case)

(This is the final version where the mistakes made by the application are allready manually corrected by me)

//////////////////////////////////////////////////////////////////////////////    ///
//
// Generated keyboard layout file with the Keyboard Layout Editor.
// For more about the software, see http://code.google.com    /p/keyboardlayouteditor
//

xkb_symbols "basic"
{
    name[Group1] = "German";
    include "latin(type4)"
    include "level3(ralt_switch)"
    include "kpdl(comma)"
key <AB01> { [              y,              Y, guillemotright,          U203A ] }; // y Y » › 
key <AB02> { [              x,              X,  guillemotleft,          U2039 ] }; // x X « ‹ 
key <AB04> { [              v,              V, doublelowquotemark, singlelowquotemark ] }; // v V „ ‚ 
key <AB05> { [              b,              B, leftdoublequotemark, leftsinglequotemark ] }; // b B “ ‘ 
key <AB06> { [              n,              N, rightdoublequotemark, rightsinglequotemark ] }; // n N ” ’ 
key <AB08> { [          comma,      semicolon,     U030C,       multiply ] }; // , ; Dˇ × 
key <AB09> { [         period,          colon,  U0323,       division ] }; // . : D? ÷ 
key <AB10> { [          minus,     underscore,          U0331,         emdash ] }; // - _ ̱ — 
key <AC02> { [              s,              S,          U017F,          U1E9E ] }; // s S ſ ẞ 
key <AC06> { [       h,       H,          U032E                 ] }; //   ̮ 
key <AC07> { [              j,              J,  dead_belowdot,  dead_abovedot ] }; // j J D? D˙ 
key <AC10> { [     odiaeresis,     Odiaeresis,  U0307,  dead_belowdot ] }; // ö Ö D˙ D? 
key <AC11> { [     adiaeresis,     Adiaeresis,    U0304,     dead_caron ] }; // ä Ä Dˉ Dˇ 
key <AD03> { [              e,              E,       EuroSign,       EuroSign ] }; // e E € € 
key <AD06> { [              z,              Z,      leftarrow,            yen ] }; // z Z ← ¥ 
key <AD11> { [     udiaeresis,       Udiaeresis, dead_diaeresis, dead_abovering ] }; // ü  D¨ D˚ 
key <AD12> { [           plus,       asterisk,     asciitilde,         macron ] }; // + * ~ ¯ 
key <AE02> { [           KP_2,       quotedbl,    twosuperior,      oneeighth ] }; // 2 " ² ⅛ 
key <AE03> { [           KP_3,        section,  threesuperior,       sterling ] }; // 3 § ³ £ 
key <AE04> { [           KP_4,         dollar,     onequarter,       currency ] }; // 4 $ ¼ ¤ 
key <AE11> { [         ssharp,       question,      backslash,   questiondown ] }; // ß ? \ ¿ 
key <AE12> { [     dead_acute,     dead_grave,          U02BF,    dead_ogonek ] }; // D´ D` ʿ D˛ 
key <BKSL> { [     numbersign,     apostrophe,          U02BE,     dead_breve ] }; // # ' ʾ D˘ 
key <LSGT> { [           less,        greater,            bar, dead_belowmacron ] }; // < > | Dˍ 
key <TLDE> { [ dead_circumflex,         degree,          U2032,          U2033 ] }; // D^ ° ′ ″ 
};

Answer 1

Update: Found a solution myself

I found a solution myself and I'm gonna post it here since someone else might profit from it

Install KeyboardLayoutEditor and edit the keyboard layout-file

1. I Downloaded and installed KeyboardLayoutEditor from https://github.com/simos/keyboardlayouteditor

   • When I tried to run it, I noticed that some packages were missing which I had to install first and which were not listed among the requirements.

2. Run KeyboardLayoutEditor and load a suitable language layout-file (de in my case)

   • Then choose the variant you want to edit (in my case "basic" since I figured I might have made some error while adding my own variant)

3. Then Click on the "Start Charakter Map" Button which will load Gucharmap (I had to install it first via Ubuntu Software Center for this to work)

4. Now choose the character you want to add to a certain key (you can also search for signs in Gucharmap by entering Unicode-Codes)

5. And drag and drop it to one of the 4 positions of a suitable key in KeyboardLayoutEditor (see below for the final code)

   • BTW I think the application made some mistakes here. Because with dragging some characters to a certain key, it didn't write the correct unicode-code into the de-file but some name of the corresponding deadkey character. (For example it wrote "dead_belowdot" instead of U0323 allthough I chose the latter character in Gucharmap and dragged it over. I suppose this is a bug.
   • The result was
     • that with these deadkey special characters I had to type them FIRST and THEN the letter I wanted them to appear over or under.
     • the other inserted characters with the correct unicode-code could be typed by FIRST typing the letter and THEN the special character I wanted to add above or below. This is the behaviour I am used to, so I tried to fix the deadkey versions:
   • My solution was to edit the de-file by hand replacing all the wrongly inserted deadkey-versions with the correct unicode-code for the combined character-version. Now I can type a letter and then add the special character.

6. When you inserted all characters you need you can save the file

   • In my case KeyboardLayoutEditor didn't just save the changes to the "basic" variant inside the "de-file" but overwrote the whole "de-file" thereby deleting all other variants
   • Allthough I think I don't need the other variants I didn't like that. So I copied the code KeyboardLayoutEditor wrote in the de-File to my de_backup replacing the "basic" section there and then saved de_backup as "de" replacing the old "de".
   • the evdev.xml doesn't have to be edited this time because an existing variant was edited so no new variant/layout has to be added to the file.

7. Then I used the sudo dpkg-reconfigure xkb-data command in the console

8. Finally I did a reboot

Result: Everythin works like a charm now! :)

Some thoughts

I don't know why it worked this time by using KeyboardLayoutEditor.

1. My first guess is that it worked because I didn't try to add a new variant to the keyboard-Layout-file (de) but edited an existing one thereby avoiding some possible mistakes with editing the de-file and the evdev-xml

2. My second guess is that it worked because KeyboardLayout editor wrote the Unicode-Codes different than I had when first manually editing the de-file.

   • For example the unicode-code I inserted looked like this: U+02BE
   • The unicode-code written by the application looks like this: U02BE
   • Perhaps by leaving out the "+" it worked? As I am a noob I don't know.
   • Some knowledgable people might see a a differend cause in the code below.

Update: According to Gunnar Hjalmarsson, my second guess may likely be correct the problem being the + signs

The Code which worked that I inserted in the keyboard layout-file (de in my case)

(This is the final version where the mistakes made by the application are allready manually corrected by me)

//////////////////////////////////////////////////////////////////////////////    ///
//
// Generated keyboard layout file with the Keyboard Layout Editor.
// For more about the software, see http://code.google.com    /p/keyboardlayouteditor
//

xkb_symbols "basic"
{
    name[Group1] = "German";
    include "latin(type4)"
    include "level3(ralt_switch)"
    include "kpdl(comma)"
key <AB01> { [              y,              Y, guillemotright,          U203A ] }; // y Y » › 
key <AB02> { [              x,              X,  guillemotleft,          U2039 ] }; // x X « ‹ 
key <AB04> { [              v,              V, doublelowquotemark, singlelowquotemark ] }; // v V „ ‚ 
key <AB05> { [              b,              B, leftdoublequotemark, leftsinglequotemark ] }; // b B “ ‘ 
key <AB06> { [              n,              N, rightdoublequotemark, rightsinglequotemark ] }; // n N ” ’ 
key <AB08> { [          comma,      semicolon,     U030C,       multiply ] }; // , ; Dˇ × 
key <AB09> { [         period,          colon,  U0323,       division ] }; // . : D? ÷ 
key <AB10> { [          minus,     underscore,          U0331,         emdash ] }; // - _ ̱ — 
key <AC02> { [              s,              S,          U017F,          U1E9E ] }; // s S ſ ẞ 
key <AC06> { [       h,       H,          U032E                 ] }; //   ̮ 
key <AC07> { [              j,              J,  dead_belowdot,  dead_abovedot ] }; // j J D? D˙ 
key <AC10> { [     odiaeresis,     Odiaeresis,  U0307,  dead_belowdot ] }; // ö Ö D˙ D? 
key <AC11> { [     adiaeresis,     Adiaeresis,    U0304,     dead_caron ] }; // ä Ä Dˉ Dˇ 
key <AD03> { [              e,              E,       EuroSign,       EuroSign ] }; // e E € € 
key <AD06> { [              z,              Z,      leftarrow,            yen ] }; // z Z ← ¥ 
key <AD11> { [     udiaeresis,       Udiaeresis, dead_diaeresis, dead_abovering ] }; // ü  D¨ D˚ 
key <AD12> { [           plus,       asterisk,     asciitilde,         macron ] }; // + * ~ ¯ 
key <AE02> { [           KP_2,       quotedbl,    twosuperior,      oneeighth ] }; // 2 " ² ⅛ 
key <AE03> { [           KP_3,        section,  threesuperior,       sterling ] }; // 3 § ³ £ 
key <AE04> { [           KP_4,         dollar,     onequarter,       currency ] }; // 4 $ ¼ ¤ 
key <AE11> { [         ssharp,       question,      backslash,   questiondown ] }; // ß ? \ ¿ 
key <AE12> { [     dead_acute,     dead_grave,          U02BF,    dead_ogonek ] }; // D´ D` ʿ D˛ 
key <BKSL> { [     numbersign,     apostrophe,          U02BE,     dead_breve ] }; // # ' ʾ D˘ 
key <LSGT> { [           less,        greater,            bar, dead_belowmacron ] }; // < > | Dˍ 
key <TLDE> { [ dead_circumflex,         degree,          U2032,          U2033 ] }; // D^ ° ′ ″ 
};