Both the SIOCSIWENCODEEXT and SIOCSIWGENIE errors are usually intimately associated with WPA/wpa_supplicant failure.
Have you tried recompiling the latest RT2790 driver with *wpa_supplicant* support enabled -- by default it's NOT!. Also, note that it only supports WPA PSK for adhoc (see line 280 of README_STA below)
- Get the latest driver, download, extract,
cd to the directory.
Inside os/linux/config.mk, find lines 10-14, which read:
# Support Wpa_Supplicant
HAS_WPA_SUPPLICANT=n
# Support Native WpaSupplicant for Network Maganger
HAS_NATIVE_WPA_SUPPLICANT_SUPPORT=n
Change the n at the end of both those lines to y.
make and sudo make install the driver (you'll need kernel headers, etc - comment if you need more info on the process).
Reboot, and see if things are any different (errors, etc.) with WPA adhoc.
If not:
- Following the
README_STA file in the extracted directory, copy it cp RT2860STA.dat /etc/Wireless/RT2860STA/RT2860STA.dat
Open that file, and change:
NetworkType=Adhoc
Authmode=WPANONE : important only this (WPA PSK) is supported for adhoc
- Try
EncrypType=NONE (and even TKIP and AES, because README does not say which is supported for adhoc)
- Set
WPAPSK to the key -- ***wpa_supplicant* may not be fully supported in adhoc*
Unload/load the the module to make it take effect: rmmod and modprobe, respectively.
See the README_STA file for more parameters and details, and try experimenting. Comment if you need more help.