X is an application that manage one or more graphics displays and one or more input devices (keyboard, mouse, etc.) connected to the computer.
It works as a server, i.e. other applications, running on the same computer or on other computer on the network, can communicate with the X server to display graphical interfaces and receive input from the user, using the X server services.
Worth to know, a common component used with an X server is the Window Manager, an application that manage the decorative elements of windows (title bars, minimize and close buttons), the resizing and moving of windows.
The X server can be started from user with startx, or more commonly can be started from a display manager, as gdm.
~/.xinitrc is a shell script used by xinit, that start the X server when not using a display manager, to define some application to start automatically in the X server.
/etc/X11/xorg.conf is a configuration file used to give the X server information about the hardware components used, but now the X server can avoid using it, because it is capable of autoconfiguring itself.