Can't send ENTER key in Vim and ncurses apps under Byobu and Tmux

Question

I recently upgraded from Ubuntu 18.04 to Ubuntu 20.04 and got a strange problem with byobu, tmux, Vim and ncurses applications: Enter doesn't work, and I need to use CTRL+M instead.

For example, if I edit a file in Vim inside a Byobu (or Tmux) window in a Gnome-Terminal, typing Enter doesn't insert a newline character, but inserts a new line with a M character. To insert a new line I need to type CTRL+M. AS a result, to exit Vim I need to type ESC :q CTRL+M.

This happens also in ncurses applications such as ncdu: instead of Enter I need to use CTRL+M.

Changing the TERM environment variable seems to help, but not 100%. The default value I get in byobu and tmux is TERM=screeen-256color, but if I change it to TERM=xterm-256color everything seems to work fine in Vim, but not in ncdu.

In addition to this, Enter key doesn't work when I use F9 to change Byobu configuration - as I said above, I need to use CTRL+M to select items in the configuration.

I tried also to add the following line in both ~/.tmux.conf and ~/.byobu/.tmux.conf, but it helped only with Vim:

set -g default-terminal "xterm-256color"

The problem happens on xterm also.

How can I solve this?

Update

I tried again with a new user, and experienced the same problem, so my guess is that it could be something system-related.

Update 2

Tinkering around with xev I found that there's something odd with this laptop's keyboard - it's an ANSI keyboard even though I'm in Italy (standard Italian keyboard layouts are ISO), and Enter button sends the wrong keycode.

When I press the Enter button it sends a kyecode 104, KP_Enter, instead of usual keycode 36, Return. So it's an hardware problem, perhaps unrelated to Ubuntu itself. Not sure why I didn't realise it before.

Answer 1

Tinkering around with xev I found that Enter button on this keyboard sends the wrong keycode. I'm leaving here my findings and how I solved, so it can be useful to others.

When I press the Enter button on this keyboard it sends a keycode 104, KP_Enter, instead of usual keycode 36, Return:

$ xev
...
KeyPress event, serial 47, synthetic NO, window 0x3c00001,
    root 0x7a5, subw 0x0, time 5065933, (762,538), root:(934,702),
    state 0x0, keycode 104 (keysym 0xff8d, KP_Enter), same_screen YES,
"   XLookupString gives 1 bytes: (0d) "
"   XmbLookupString gives 1 bytes: (0d) "
    XFilterEvent returns: False
...

So it's an hardware "problem", since Return and KP_Enter are not the same and don't give always the same results.

So to solve this problem I need to remap keycode 104 to Return. This can be don using xmodmap like this:

$ xmodmap -e "keycode 104 = Return NoSymbol Return"

To make the change permanent, this command can be run at every login adding it to "Startup Applications", like this:

Answer 2

That's very odd. It is working outside tmux, yes?

Does it still happen if you run tmux without a config? Start a new tmux with tmux -Ltest -f/dev/null new and see if the problem exists with it too. If not then you will need to look at your config and see what it causing it.

The only other things I can think are to check if your terminal is actually sending C-m for Enter (you can run cat then press the key and see what is sent), or something to do with the kent capability in the terminfo entries. But they would be fiddly to test and why should they be wrong on Ubuntu anyway and just for you? Maybe try making sure your ncurses packages are up to date.