0

This is about my successful experience in bringing up Ubuntu 17.10 x86_64 on the clang-built Linux 4.15.7 kernel(latest stable as of now) and proprietary Nvidia 390.25(latest as of now) driver for my Nvidia GTX GeForce 860M graphics card.

The problem

Linux kernel 4.15.7 was built using clang as the host and target compiler and installed along with the modules based on default kernel config. I had to disable exofs module as it had non-standard "VLAs embedded within structs" that clang doesn't support. Then I also compiled Nvidia 390.25 driver source after extracting it from its propitiatory installer - I had to ignore the compiler version mismatch psuedo-warning as kernel headers included the thread model and also the platform information while Nvidia driver script check resulted in just the version.

After installing thus built Nvidia driver modules, I rebooted my system, the Ubuntu 17.10 booting didn't complete - it got stuck at Loading ... 4.15.7.

To further isolate the compatibility issues (if any) of kernel with gcc/clang vs Nvidia driver with gcc/clang, Compiled Linux kernel 4.15.7 with gcc and Nvidia 390.25 driver with clang, and vice versa. For the first, the booting process passed the Loading ... 4.15.7 phase and moved onto some systemd service, but got stuck there. For the second, again, it got stuck in the Loading ... 4.15.7 phase again. The system journalctl/logs didn't help much.

Improve this question
4
  • 1
    Well, it's in Q&A format for the ones who have question regarding bringup of clang-built kernel and kernel modules for Ubuntu OS. And it's not off-topic like some totally unrelated post which I see plenty here and elsewhere and yet get retained for discussion. So, I don't think this needs to be closed, etc.
    – user194850
    Apr 7, 2018 at 14:19
  • 1
    Well, report covers many topics with respect to clang/LLVM and Linux kernel/modules/Ubuntu compatibility. I don't see people have tried these things at all so far. So, Dobey, I definitely think my report addresses many questions and answers that like-minded people will have. It's not a blog post either.
    – user194850
    Apr 7, 2018 at 14:38
  • 1
    As mentioned in my other post, it's a report which involves questions, answers, analysis, investigation, results, conclusion and more. So, this fits inside Q&A category.
    – user194850
    Apr 7, 2018 at 15:57
  • 7
    Then please edit your questions and turn them into actual questions, and not simply "reports" as you are even calling them. This is not a blog posting site, an issue tracker, or a forum. Though, even written as a proper question, this is still unlikely to be relevant to general user base, as it requires manually rebuilding core parts of the system with a different compiler, for the issue to exist.
    – dobey
    Apr 7, 2018 at 17:45

1 Answer 1

Reset to default
0

Stage 1: Having worked with Qemu/Kvm earlier, I wanted to get hold of the actual failing point in bringing the clang-built kernel up. So, I tried to bring up the clang-built Linux kernel 4.15.7 image using the x86_64 default config to begin with and also clang-built busybox x86_64 for the root file system (initrd), on Qemu/kvm x86_64. This worked. I got the shell and busybox utilities were working and so on.

On host Ubuntu 17.10 x86_64:

$ strings -a defconfig-linux-4.15.7/linux-4.15.7/vmlinux | grep "clang version" | head -1
Linux version 4.15.7 (exp@exp) (clang version 5.0.0-3 (tags/RELEASE_500/final)) #1 SMP Fri Mar 2 21:15:24 PST 2018

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 17.10
Release:    17.10
Codename:   artful

$ clang -v
clang version 5.0.0-3 (tags/RELEASE_500/final)
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

Within Qemu/kvm guest, Linux kernel 4.15.7(x86_64 defconfig)/Busybox 64-bit :

/ # cat /proc/version 
Linux version 4.15.7 (exp@exp) (clang version 5.0.0-3 (tags/RELEASE_500/final)) #1 SMP Fri Mar 2 21:15:24 PST 2018
/ # 

/ # uname -arv
Linux (none) 4.15.7 #1 SMP Fri Mar 2 21:15:24 PST 2018 x86_64 GNU/Linux
/ # 
/ # 
/ # strings -a bin/busybox | grep "clang version"
clang version 5.0.0-3 (tags/RELEASE_500/final)
/ # 
/ # 
/ # dmesg | grep QEMU
[    0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.9.3-0-ge2fc41e-prebuilt.qemu-project.org 04/01/2014
[    0.024000] smpboot: CPU0: Intel QEMU Virtual CPU version 2.5+ (family: 0x6, model: 0x6, stepping: 0x3)
[    0.337415] ata2.00: ATAPI: QEMU DVD-ROM, 2.5+, max UDMA/100
[    0.339072] scsi 1:0:0:0: CD-ROM            QEMU     QEMU DVD-ROM     2.5+ PQ: 0 ANSI: 5
/ # 
/ #

Stage 2: With the above initial success with defconfig kernel, I took the host Ubuntu 17.10 kernel config and rebuilt the Linux kernel 4.15.7 image. With this I kept rest of the busybox the same and tried to bring them up on Qemu/kvm as earlier - this didn't work. Resulted in general protection faults and kernel oops - kernel panic. I looked into the kernel stack dump.

I noticed the first issue was within kernel irq work tick logic. Then the next issue was with kernel live instruction update logic. I worked on both of them. Rebuilt the kernel. Tried again with Qemu/kvm - got the shell, no panic this time. Next step was to install kernel modules and Nvidia driver modules as well. I did that but this time, with all the Ubuntu kernel modules and Nvidia driver module, Qemu/kvm showed some colored pixels and got stuck - mostly due the difference between host's hardware and guest's virtualized hardware. So, I wanted to try only with Nvidia driver as it was of prime concern as it's the one that I use on my host for the graphics. So, I included (didn't install) Nvidia driver modules in the busybox rootfs and tried to bring up Qemu/kvm as usual - this time, there was some kernel panic due to rootfs-not-found though there was a rootfs.

So, I tried to compress all of the Nvidia driver modules onto a zip and include it in the rootfs. But, when I tried to extract the zip with the Qemu/kvm guest, there was no space left on device. So, I tried to construct a 10G virtual qemu disk and install the rootfs within that disk to address the space concern - but, after a while I called this off as I was more interested in getting the clang-built Nvidia driver on real system hardware than on the emulated Qemu/kvm environment (I know it doesn't have the real Nvidia graphics card available via GPU-passthrough so that it can test the clang-built Nvidia 390.25 driver on it before I could try that on host's real system hardware/graphics card, but gave it a shot anyway just to see if clang-built Nvidia driver exhibits any issues within Qemu/kvm when loaded).

Within Qemu/kvm guest, Linux kernel 4.15.7(host Ubuntu 17.10 x86_64 config)/Busybox64-bit:

/ # cat /proc/version 
Linux version 4.15.7 (exp@exp) (clang version 5.0.0-3 (tags/RELEASE_500/final)) #20 SMP Sat Mar 3 20:46:15 PST 2018
/ # 
/ # 
/ # uname -arv
Linux (none) 4.15.7 #20 SMP Sat Mar 3 20:46:15 PST 2018 x86_64 GNU/Linux
/ # 
/ # 
/ # 
/ # dmesg | grep QEMU
[    0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.9.3-0-ge2fc41e-prebuilt.qemu-project.org 04/01/2014
[    0.024000] smpboot: CPU0: Intel QEMU Virtual CPU version 2.5+ (family: 0x6, model: 0x6, stepping: 0x3)
[    0.340776] ata2.00: ATAPI: QEMU DVD-ROM, 2.5+, max UDMA/100
[    0.343135] scsi 1:0:0:0: CD-ROM            QEMU     QEMU DVD-ROM     2.5+ PQ: 0 ANSI: 5
/ # 
/ # strings -a bin/busybox | grep "clang version"
clang version 5.0.0-3 (tags/RELEASE_500/final)
/ #

Stage 3 and Success: As my main and actual target was my host (Ubuntu 17.10), I installed this clang-built kernel (with above issues worked upon) and Nvidia 390.25 drivers on my host system directly. Rebooted my system. Bam! I got my Ubuntu 17.10 x86_64 up and running with clang-built 4.15.7 kernel and clang-built Nvidia 390.25 driver without any issues.

On host Ubuntu 17.10 x86_64 with clang-built Linux kernel 4.15.7 and clang-built Nvidia 390.25 GeForce GTX 860M proprietary driver - my successful research and experience : 

$ dmesg | grep clang
[    0.000000] Linux version 4.15.7 (exp@exp) (clang version 5.0.0-3 (tags/RELEASE_500/final)) #20 SMP Sat Mar 3 20:46:15 PST 2018

$ cat /proc/version 
Linux version 4.15.7 (exp@exp) (clang version 5.0.0-3 (tags/RELEASE_500/final)) #20 SMP Sat Mar 3 20:46:15 PST 2018

$ uname -arv
Linux exp 4.15.7 #20 SMP Sat Mar 3 20:46:15 PST 2018 x86_64 x86_64 x86_64 GNU/Linux

$ nvidia-smi
Sun Mar  4 18:15:39 2018       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 390.25                 Driver Version: 390.25                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 860M    Off  | 00000000:01:00.0 Off |                  N/A |
| N/A   54C    P0    N/A /  N/A |    370MiB /  4046MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

exp@exp:~$ dmesg | grep NVIDIA | grep 390.25
[   17.643496] NVRM: loading NVIDIA UNIX x86_64 Kernel Module  390.25  Wed Jan 24 20:02:43 PST 2018 (using threaded interrupts)
[   17.689835] nvidia-modeset: Loading NVIDIA Kernel Mode Setting Driver for UNIX platforms  390.25  Wed Jan 24 19:29:37 PST 2018

$ lsmod | grep nvidia
nvidia_uvm            815104  0 
nvidia_drm             24576  2 
nvidia_modeset       1097728  5 nvidia_drm
nvidia              14352384  334 nvidia_uvm,nvidia_modeset
ipmi_msghandler        61440  2 nvidia,ipmi_devintf
drm                   454656  6 nvidia_drm,i915,drm_kms_helper

# clang-built Linux kernel 4.15.7 image and Nvidia 390.25 driver modules having clang compiler specific
# metadata within them.
#

$ eclang /lib/modules/4.15.7/build/vmlinux | grep "Linux"
++ strings -a /lib/modules/4.15.7/build/vmlinux
++ grep clang
Linux version 4.15.7 (exp@exp) (clang version 5.0.0-3 (tags/RELEASE_500/final)) #20 SMP Sat Mar 3 20:46:15 PST 2018
exp@exp:~$ eclang /lib/modules/4.15.7/kernel/drivers/video/nvidia.ko | head -1
++ strings -a /lib/modules/4.15.7/kernel/drivers/video/nvidia.ko
++ grep clang
clang version 5.0.0-3 (tags/RELEASE_500/final)

$ eclang /lib/modules/4.15.7/kernel/drivers/video/nvidia-drm.ko | head -1
++ strings -a /lib/modules/4.15.7/kernel/drivers/video/nvidia-drm.ko
++ grep clang
clang version 5.0.0-3 (tags/RELEASE_500/final)

$ eclang /lib/modules/4.15.7/kernel/drivers/video/nvidia-uvm.ko | head -1
++ strings -a /lib/modules/4.15.7/kernel/drivers/video/nvidia-uvm.ko
++ grep clang
clang version 5.0.0-3 (tags/RELEASE_500/final)

$ eclang /lib/modules/4.15.7/kernel/drivers/video/nvidia-modeset.ko | head -1
++ strings -a /lib/modules/4.15.7/kernel/drivers/video/nvidia-modeset.ko
++ grep clang
clang version 5.0.0-3 (tags/RELEASE_500/final)

# DMI specific hardware information - verifies and confirms the physical hardware used
# instead of QEMU or any other virtual platform for running clang-built kernel and
# Nvidia 390.25 driver.
#

$ sudo dmidecode -t bios | grep Vendor
    Vendor: LENOVO

$ sudo dmidecode -t system | grep "Manufacturer:\|Version"
    Manufacturer: LENOVO
    Version: Lenovo Y50-70 Touch

$ sudo dmidecode -t processor | grep Version
    Version: Intel(R) Core(TM) i7-4710HQ CPU @ 2.50GHz

NOTE 1: For all the detailed logs (from each stage), you can read my answer for LLVM-Clang/Linux kernel question here :Building Linux kernel with LLVM/clang.

NOTE 2: clang's warning are more pedantic than gcc as I saw. But, for the time being, to get the kernel/nvidia driver up and running, I have ignored them to see if they are not serious (so far they are not).

UPDATE 1 : For the sake of universality, I repeated the above with subsequent kernel versions (4.15.10 and 4.15.11 which is latest as of mid-March 2018) and latest Nvidia graphics GTX GeForce driver v390.42 which is latest as of mid-March 2018 - both of them built using clang as earlier after updating kernel 4.15.11 code in IRQ work tick logic and live instruction update logic as I did above for 4.15.7, and finally installed both of them on my system hardware directly (no Qemu/kvm experiments this time as the earlier 4.15.7 kernel had come up successfully after working upon IRQ work tick logic and live instruction update logic). With that, again, Ubuntu 17.10 x86_64 successfully came up on my system hardware, on clang-built kernel 4.15.11 and clang-built latest Nvidia driver v390.42, as it did in the earlier case of kernel 4.15.7 and Nvidia driver v390.25.

Below are the details for the Ubuntu 17.10 x86_64 running on clang-built latest v4.15.11 kernel and clang-built latest Nvidia driver v390.42 scenario:

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 17.10
Release:    17.10
Codename:   artful

$ cat /proc/version 
Linux version 4.15.11 (exp@exp) (clang version 5.0.0-3 (tags/RELEASE_500/final)) #2 SMP Tue Mar 20 05:37:06 PDT 2018

$ eclang /lib/modules/4.15.11/build/vmlinux | grep Linux
++ strings -a /lib/modules/4.15.11/build/vmlinux
++ grep clang
Linux version 4.15.11 (exp@exp) (clang version 5.0.0-3 (tags/RELEASE_500/final)) #2 SMP Tue Mar 20 05:37:06 PDT 2018

$ sudo dmidecode -t system | grep "Manufacturer:\|Version"
    Manufacturer: LENOVO
    Version: Lenovo Y50-70 Touch

$ dmesg | grep NVIDIA
[   19.491452] nvidia: module license 'NVIDIA' taints kernel.
[   19.499360] NVRM: loading NVIDIA UNIX x86_64 Kernel Module  390.42  Sat Mar  3 04:10:22 PST 2018 (using threaded interrupts)
[   19.542830] nvidia-modeset: Loading NVIDIA Kernel Mode Setting Driver for UNIX platforms  390.42  Sat Mar  3 03:30:48 PST 2018

$ nvidia-smi
Tue Mar 20 22:15:00 2018       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 390.42                 Driver Version: 390.42                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 860M    Off  | 00000000:01:00.0 Off |                  N/A |
| N/A   52C    P8    N/A /  N/A |    451MiB /  4046MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

$ glxinfo | grep NVIDIA
server glx vendor string: NVIDIA Corporation
client glx vendor string: NVIDIA Corporation
OpenGL vendor string: NVIDIA Corporation
OpenGL core profile version string: 4.5.0 NVIDIA 390.42
OpenGL core profile shading language version string: 4.50 NVIDIA
OpenGL version string: 4.6.0 NVIDIA 390.42
OpenGL shading language version string: 4.60 NVIDIA
OpenGL ES profile version string: OpenGL ES 3.2 NVIDIA 390.42

$ lsmod | grep nv
nvidia_uvm            815104  0 
nvidia_drm             24576  2 
nvidia_modeset       1105920  5 nvidia_drm
nvidia              14368768  334 nvidia_uvm,nvidia_modeset
ipmi_msghandler        61440  2 nvidia,ipmi_devintf
drm                   454656  6 nvidia_drm,i915,drm_kms_helper

$ ~/nvidia-driver-compiler.sh 
++ head -1
+++ uname -r
++ eclang /lib/modules/4.15.11/kernel/drivers/video/nvidia.ko
+++ strings -a /lib/modules/4.15.11/kernel/drivers/video/nvidia.ko
+++ grep clang
clang version 5.0.0-3 (tags/RELEASE_500/final)
++ head -1
+++ uname -r
++ eclang /lib/modules/4.15.11/kernel/drivers/video/nvidia-drm.ko
+++ strings -a /lib/modules/4.15.11/kernel/drivers/video/nvidia-drm.ko
+++ grep clang
clang version 5.0.0-3 (tags/RELEASE_500/final)
+++ uname -r
++ head -1
++ eclang /lib/modules/4.15.11/kernel/drivers/video/nvidia-modeset.ko
+++ strings -a /lib/modules/4.15.11/kernel/drivers/video/nvidia-modeset.ko
+++ grep clang
clang version 5.0.0-3 (tags/RELEASE_500/final)
+++ uname -r
++ head -1
++ eclang /lib/modules/4.15.11/kernel/drivers/video/nvidia-uvm.ko
+++ strings -a /lib/modules/4.15.11/kernel/drivers/video/nvidia-uvm.ko
+++ grep clang
clang version 5.0.0-3 (tags/RELEASE_500/final)

$ ~/nvidia_driver_modules_version.sh 
++ grep '^version='
+++ uname -r
++ strings -a /lib/modules/4.15.11/kernel/drivers/video/nvidia.ko
version=390.42
++ grep '^version='
+++ uname -r
++ strings -a /lib/modules/4.15.11/kernel/drivers/video/nvidia-drm.ko
version=390.42
++ grep '^version='
+++ uname -r
++ strings -a /lib/modules/4.15.11/kernel/drivers/video/nvidia-uvm.ko
+++ uname -r
++ grep '^version='
++ strings -a /lib/modules/4.15.11/kernel/drivers/video/nvidia-modeset.ko
version=390.42
Improve this answer
1
  • 1
    Raghavan, I always leave a comment before I downvote, so here goes: 1. TL;DR 2. lots of superfluous information: Please don't tell users why you did it that way (except maybe a one-line at the bottom) nor what the result is, just what you did with the exact commands. Please edit both your Q and A (tried to have a go at your Q to actually make it a Q, but not deleting 80% of your "answer" as there would not be enough left to actually make it an answer. Please review and ping me @Fabby and I'll come back and upvote.
    – Fabby
    Apr 7, 2018 at 19:26

You must log in to answer this question.