Since nautilus is written in C, how is the GUI created?
Is there a specific C framework that allows for GUI creation or is it done some other way?
Nautilus is written in C mostly, as evident from the project's github page:
According to Wikipedia it uses GTK3 as of version 2.32.
If you want to create GUI with C, there's not much choice - GTK is the best option ( although I'll hint at something known as Borland's Turbo C which is like straight from the 1980s). I'd suggest you move up to C++. With C++ there's GTK and Qt options.