I wish I would have seen this question earlier. I'm sorry that you have been affected for a long time. :-(
This is a known issue (please refer to https://github.com/LIS/lis-next/issues/655) which has been fixed in the Linux mainline kernel since last Nov (https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=5f1251a48c17b54939d7477305e39679a565382c).
For Ubuntu 20.04, as I just checked, the latest linux-azure kernel Ubuntu-azure-5.4.0-1039.41 (Jan 18) still does not have the fix, but the generic 5.4 kernel Ubuntu-5.4.0-66.74 and the HWE kernel Ubuntu-hwe-5.8-5.8.0-44.50_20.04.1 already have the fix. You probably want to upgrade to either of the two kernels that have the fix.
If you can't upgrade your kernel immediately, there is a workaround: please blacklist the Hyper-V synthetic framebuffer driver (the file location can be found by "modinfo hyperv_fb"), and then Linux will automatically use the legacy vesafb driver (if it’s a Generation-1 VM) or the efifb driver (if it’s a Generation-2 VM); if it’s a Generation-1 VM, please also add the kernel parameter "video=vesafb:mtrr:3", which tells the legacy framebuffer driver "vesafb" to map the legacy framebuffer cacheable. The rationale of the workaround is that the legacy vesafb/efifb drivers are fast since they map (or can be instructed to map) the framebuffer cacheable.