I'm creating a file server with Ubuntu Server 16.04 LTS. The motherboard doesn't have a on-board graphics card so I'm using a Nvidia graphics card for the install. I've tested that I can install with this graphics card using the free nouveau drivers. I'm concerned about system stability. I know this same graphics card causes serious problems in Ubuntu Desktop when not using the non-free drivers.
So since I don't have a desktop or X installed, and will not be interacting with the machine locally, should I leave the default driver or install the non-free Nvidia driver?
I'm asking because I'm not sure if graphics card driver problems are restricted to X and desktop environments or if they are always in play and just become a bigger problem when working in desktop environments. Plus I don't want to install the non-free Nvidia drivers because I'm not sure how to prevent it from also installing a graphical desktop.
Thanks for any guidance here.