I just downloaded the developer driver from Nvidia's CUDA Zone. Executed it and typed in restart gdm and the it worked. However after rebooting X didn't come up (xorg log), to fix it I can simply rerun the installer and the restart gdm again.
What I don't understand is where it breaks and why X11 doesn't find the module?! I removed all (I think..) of the Nvidia packages which can with Ubuntu 10.10. I've also googled a bit and found some HOWTOs about manually installing the Nvidia driver. However all of them were for an earlier version of Ubuntu.
Please do not respond telling me I should use the proprietary Nvidia driver supplied by Ubuntu, because I need the development drivers.