I feel your pain. There is a ton of information around, and most of it is close, but not exactly what you need. I recently succeeded in building 20.04 kernels for the pi 4. My notes are attached here. I'm building the 32-bit version, and I'm not cross-compiling; I'm building on the pi itself.
To customize, edit files under .../debian.raspi/config/
start with a 32GB card (might just barely fit on a 16GB card)
download, install, and update 32-bit (= "armhf") Ubuntu Server 20.04 for Raspberry Pi (preinstalled image)
uncomment the following 2 lines in /etc/apt/sources.list:
deb-src http://ports.ubuntu.com/ubuntu-ports focal main restricted
deb-src http://ports.ubuntu.com/ubuntu-ports focal-updates main restricted
sudo apt update
sudo apt-get build-dep linux linux-image-$(uname -r) # brings in nearly a gigabyte
sudo apt install libncurses-dev linux-tools-common fakeroot
# these notes assume your build tree will be in ~/kbuild.
# if you'd prefer to put it elsewhere, simply substitute your location for ~/kbuild in all that follows.
# NOTE: the following git command retrieves the *raspi* branch!
git clone --depth=1 git://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/focal -b raspi
# intial build - takes 2 hours 20 minutes, creates 7 .deb files in ~/kbuild
fakeroot debian/rules clean
fakeroot debian/rules binary-headers binary binary-perarch
# install and run the newly-built kernel:
sudo apt install libunwind8 linux-tools-common libdw1
sudo dpkg -i *.deb
sync; sudo reboot
# rebuild - takes 30 minutes
rm -rf debian/linux-libc-dev/usr/include/arm-linux-gnueabihf
rm -rf debian/build/build-raspi/_____________________________________________dkms
# if you've changed any header files, add binary-headers to the following line:
fakeroot debian/rules binary binary-perarch