There are some "Special" directories in your home directory, Downloads are one of them.
=== Command line / manual method ===
it should look like this:
# This file is written by xdg-user-dirs-update
# If you want to change or add directories, just edit the line you're
# interested in. All local changes will be retained on the next run
# Format is XDG_xxx_DIR="$HOME/yyy", where yyy is a shell-escaped
# homedir-relative path, or XDG_xxx_DIR="/yyy", where /yyy is an
# absolute path. No other format is supported.
Delete or comment out ( add a
# at the start) the
Then delete ~/Downloads
If that fails, install and run Ubuntu Tweak