I don't see any differences really, especially if you are only looking for things that "cannot be addressed by configuring the machine with a reasonable amount of effort"
They are both built on the same base, so start the same.
They include different packages and installers, but these differences are easily changed by configuration/package changes post-install.
Ubuntu Server uses
subiquity as its installer, which allows the installer to select to use GA or HWE kernel stacks, whereas with Ubuntu Desktop that choice is made by the ISO used to install (before Focal anyway; now it's done post-install if you want a change; refer to documentation).
subiquity is more modern and allows for greater choice (when compared with
ubiquity [desktop installer] which may never be used again if all goes well on its replacement), but these install options can be changed post-install, I consider, easily.
If I wanted to know the differences... I'd grab the manifest of the ISOs I wanted to compare; and compare them.
All Ubuntu ISOs are built from packages, and the manifest shows the packages included in the ISO. It's a tool we somewhat regularly use when something goes wrong (I'm thinking here of dailies but the same applies to ISOs of the official releases, which are just the last daily/RC [release candidate] anyway), to check if something changed, i.e. were different packages pulled in & thus included, maybe because of a dependency change which caused it, etc.
Ubuntu provides manifests for all ISOs.
The flaw (or limitation) with the manifests, is the installers give users options they can pick which cause the packages on the ISO to all be installed, or only some installed (and if internet is available and download is available; updated packages may mean manifest package versions are not what you'll end up with anyway).
But to me they are the same.
If I had to pick a difference, it would be that Ubuntu Server uses netplan.io, where Ubuntu Desktop will use NetworkManager. It's not a major difference but it's about all I can think of, and, along with installer differences, are easily changed anyway if so desired.
Others of course will see it differently to how I do.