I remember reading that servers do not have a GUI because X11 is a security risk. Why?
Ubuntu's community documentation explains the real reasons why it's not recommended to run a GUI on a production server system:
"ServerGUI" by "Contributors to the Ubuntu documentation wiki", reproduced here as permitted by CC-BY-SA 3.0.
Contrary to somewhat common misconception, X11 being a server really has nothing to do with why running a GUI on a production server is considered to be non-ideal from a security perspective. X11 is virtually never configured by default to be accessible over a network anymore, on any operating system. No version of Ubuntu has ever had X11 run a network-accessible server in the default configuration. (To access X11 on Ubuntu via TCP, you have to forward it though SSH or manually reconfigure the server.)
Furthermore, even if X11 did run a network-accessible server, this would not be a reason not to have it installed on a production server system. Anyone running a production server is presumably capable of configuring it for their needs and auditing it to make sure undesirable services are not running. (If they cannot, that will pose a far greater threat to their security than would be created by having a GUI installed.) Even if X11 had to have a port listening on a physical network interface (which is not the case), the port could easily be blocked by reconfiguring the built-in
In contrast, the problems listed above are not so easy to overcome by reconfiguration.
Every running process is a security risk. Particularly those which listen on a network port (X11 does).
General good practise is not to run anything on a server that doesn't absolutely need to be there, and X11 certainly doesn't need to be on a server that you'll SSH into.
I doubt the article you read was talking about a specific vulnerability in X11 (it would have been fixed if so, vulnerabilities don't tend to hang around unfixed for long), but rather just general good practise.
This is because the X Window System poses a serious security risk if it is not properly secured. An X11 "display" is the X11 server running on your desktop, and includes the screen, keyboard and mouse. If your X11 display is insecure, it will allow a program running anywhere on the Internet to connect to it and the connection may be completely invisible to you. Once connected, that program has full access to your display, which means that it can:
A thumb rule is NOT to use
A good way is to forward X is through