I feel your pain. There is a ton of information around, and most of it is close, but not exactly what you need. I recently succeeded in building and running 20.04 kernels for the pi 4. My notes are attached here. I'm not cross-compiling; I'm building on the pi itself. This has worked for me on both 32-bit and 64-bit installations.
The original git command I gave below was:
git clone --depth=1 git://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/focal -b raspi
That got me the latest code at the time I did it, but it has not been updated since. At this moment, it is still at kernel version 1009.
You can get the source for the exact kernel you currently have installed by doing:
apt-get source linux-image-$(uname -r)
If you do that, you will see a message indicating you can get the latest source code by doing this:
git clone git://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux-raspi/+git/focal
I did that just now and got source for kernel version 1012, which is later but still not the latest. The most recent system update installed binary kernel version 1013 on my pi 4.
I have updated the note below with the best information I have, but I'd still love to know if there is a publically-available git repo with even more recent pi 4 kernel source.
Here are the notes:
start with a 32GB card (might just barely fit on a 16GB card)
download, install, and update 32-bit or 64-bit Ubuntu Server 20.04 for Raspberry Pi (preinstalled image)
uncomment the following 2 lines in /etc/apt/sources.list:
deb-src http://ports.ubuntu.com/ubuntu-ports focal main restricted
deb-src http://ports.ubuntu.com/ubuntu-ports focal-updates main restricted
sudo apt update
sudo apt-get build-dep linux linux-image-$(uname -r) # brings in nearly a gigabyte
sudo apt install libncurses-dev linux-tools-common fakeroot
# these notes assume your build tree will be in ~/kbuild.
# if you'd prefer to put it elsewhere, simply substitute your location for ~/kbuild in all that follows.
git clone --depth=1 git://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux-raspi/+git/focal
# To customize, edit files under .../debian.raspi/config/
# intial build - takes 2 hours 20 minutes, creates 7 .deb files in ~/kbuild
fakeroot debian/rules clean
fakeroot debian/rules binary-headers binary binary-perarch
# install and run the newly-built kernel:
sudo apt install libunwind8 linux-tools-common libdw1
sudo dpkg -i *.deb
sync; sudo reboot
# rebuild - takes 30 minutes
rm -rf debian/linux-libc-dev/usr/include/arm-linux-gnueabihf
rm -rf debian/build/build-raspi/_____________________________________________dkms
# if you've changed any header files, add binary-headers to the following line:
fakeroot debian/rules binary binary-perarch