Stage 1: Having worked with Qemu/Kvm earlier, I wanted to get hold of the actual failing point in bringing the clang-built kernel up. So, I tried to bring up the clang-built Linux kernel 4.15.7 image using the x86_64 default config to begin with and also clang-built busybox
x86_64 for the root file system (initrd), on Qemu/kvm x86_64. This worked. I got the shell and busybox
utilities were working and so on.
On host Ubuntu 17.10 x86_64:
$ strings -a defconfig-linux-4.15.7/linux-4.15.7/vmlinux | grep "clang version" | head -1
Linux version 4.15.7 (exp@exp) (clang version 5.0.0-3 (tags/RELEASE_500/final)) #1 SMP Fri Mar 2 21:15:24 PST 2018
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 17.10
Release: 17.10
Codename: artful
$ clang -v
clang version 5.0.0-3 (tags/RELEASE_500/final)
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
Within Qemu/kvm guest, Linux kernel 4.15.7(x86_64 defconfig)/Busybox 64-bit :
/ # cat /proc/version
Linux version 4.15.7 (exp@exp) (clang version 5.0.0-3 (tags/RELEASE_500/final)) #1 SMP Fri Mar 2 21:15:24 PST 2018
/ #
/ # uname -arv
Linux (none) 4.15.7 #1 SMP Fri Mar 2 21:15:24 PST 2018 x86_64 GNU/Linux
/ #
/ #
/ # strings -a bin/busybox | grep "clang version"
clang version 5.0.0-3 (tags/RELEASE_500/final)
/ #
/ #
/ # dmesg | grep QEMU
[ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.9.3-0-ge2fc41e-prebuilt.qemu-project.org 04/01/2014
[ 0.024000] smpboot: CPU0: Intel QEMU Virtual CPU version 2.5+ (family: 0x6, model: 0x6, stepping: 0x3)
[ 0.337415] ata2.00: ATAPI: QEMU DVD-ROM, 2.5+, max UDMA/100
[ 0.339072] scsi 1:0:0:0: CD-ROM QEMU QEMU DVD-ROM 2.5+ PQ: 0 ANSI: 5
/ #
/ #
Stage 2: With the above initial success with defconfig kernel, I took the host Ubuntu 17.10 kernel config and rebuilt the Linux kernel 4.15.7 image. With this I kept rest of the busybox the same and tried to bring them up on Qemu/kvm as earlier - this didn't work. Resulted in general protection faults
and kernel oops
- kernel panic
. I looked into the kernel stack dump.
I noticed the first issue was within kernel irq work tick
logic. Then the next issue was with kernel live instruction update
logic. I worked on both of them. Rebuilt the kernel. Tried again with Qemu/kvm - got the shell, no panic this time. Next step was to install kernel modules and Nvidia driver modules as well. I did that but this time, with all the Ubuntu kernel modules and Nvidia driver module, Qemu/kvm showed some colored pixels and got stuck - mostly due the difference between host's hardware and guest's virtualized hardware. So, I wanted to try only with Nvidia driver as it was of prime concern as it's the one that I use on my host for the graphics. So, I included (didn't install) Nvidia driver modules in the busybox rootfs and tried to bring up Qemu/kvm as usual - this time, there was some kernel panic due to rootfs-not-found though there was a rootfs.
So, I tried to compress all of the Nvidia driver modules onto a zip and include it in the rootfs. But, when I tried to extract the zip with the Qemu/kvm guest, there was no space left on device
. So, I tried to construct a 10G virtual qemu disk and install the rootfs within that disk to address the space concern - but, after a while I called this off as I was more interested in getting the clang-built Nvidia driver on real system hardware than on the emulated Qemu/kvm environment (I know it doesn't have the real Nvidia graphics card available via GPU-passthrough so that it can test the clang-built Nvidia 390.25 driver on it before I could try that on host's real system hardware/graphics card, but gave it a shot anyway just to see if clang-built Nvidia driver exhibits any issues within Qemu/kvm when loaded).
Within Qemu/kvm guest, Linux kernel 4.15.7(host Ubuntu 17.10 x86_64 config)/Busybox64-bit:
/ # cat /proc/version
Linux version 4.15.7 (exp@exp) (clang version 5.0.0-3 (tags/RELEASE_500/final)) #20 SMP Sat Mar 3 20:46:15 PST 2018
/ #
/ #
/ # uname -arv
Linux (none) 4.15.7 #20 SMP Sat Mar 3 20:46:15 PST 2018 x86_64 GNU/Linux
/ #
/ #
/ #
/ # dmesg | grep QEMU
[ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.9.3-0-ge2fc41e-prebuilt.qemu-project.org 04/01/2014
[ 0.024000] smpboot: CPU0: Intel QEMU Virtual CPU version 2.5+ (family: 0x6, model: 0x6, stepping: 0x3)
[ 0.340776] ata2.00: ATAPI: QEMU DVD-ROM, 2.5+, max UDMA/100
[ 0.343135] scsi 1:0:0:0: CD-ROM QEMU QEMU DVD-ROM 2.5+ PQ: 0 ANSI: 5
/ #
/ # strings -a bin/busybox | grep "clang version"
clang version 5.0.0-3 (tags/RELEASE_500/final)
/ #
Stage 3 and Success: As my main and actual target was my host (Ubuntu 17.10), I installed this clang-built kernel (with above issues worked upon) and Nvidia 390.25 drivers on my host system directly. Rebooted my system. Bam! I got my Ubuntu 17.10 x86_64 up and running with clang-built 4.15.7 kernel and clang-built Nvidia 390.25 driver without any issues.
On host Ubuntu 17.10 x86_64 with clang-built Linux kernel 4.15.7 and clang-built Nvidia 390.25 GeForce GTX 860M proprietary driver - my successful research and experience :
$ dmesg | grep clang
[ 0.000000] Linux version 4.15.7 (exp@exp) (clang version 5.0.0-3 (tags/RELEASE_500/final)) #20 SMP Sat Mar 3 20:46:15 PST 2018
$ cat /proc/version
Linux version 4.15.7 (exp@exp) (clang version 5.0.0-3 (tags/RELEASE_500/final)) #20 SMP Sat Mar 3 20:46:15 PST 2018
$ uname -arv
Linux exp 4.15.7 #20 SMP Sat Mar 3 20:46:15 PST 2018 x86_64 x86_64 x86_64 GNU/Linux
$ nvidia-smi
Sun Mar 4 18:15:39 2018
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 390.25 Driver Version: 390.25 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 860M Off | 00000000:01:00.0 Off | N/A |
| N/A 54C P0 N/A / N/A | 370MiB / 4046MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
exp@exp:~$ dmesg | grep NVIDIA | grep 390.25
[ 17.643496] NVRM: loading NVIDIA UNIX x86_64 Kernel Module 390.25 Wed Jan 24 20:02:43 PST 2018 (using threaded interrupts)
[ 17.689835] nvidia-modeset: Loading NVIDIA Kernel Mode Setting Driver for UNIX platforms 390.25 Wed Jan 24 19:29:37 PST 2018
$ lsmod | grep nvidia
nvidia_uvm 815104 0
nvidia_drm 24576 2
nvidia_modeset 1097728 5 nvidia_drm
nvidia 14352384 334 nvidia_uvm,nvidia_modeset
ipmi_msghandler 61440 2 nvidia,ipmi_devintf
drm 454656 6 nvidia_drm,i915,drm_kms_helper
# clang-built Linux kernel 4.15.7 image and Nvidia 390.25 driver modules having clang compiler specific
# metadata within them.
#
$ eclang /lib/modules/4.15.7/build/vmlinux | grep "Linux"
++ strings -a /lib/modules/4.15.7/build/vmlinux
++ grep clang
Linux version 4.15.7 (exp@exp) (clang version 5.0.0-3 (tags/RELEASE_500/final)) #20 SMP Sat Mar 3 20:46:15 PST 2018
exp@exp:~$ eclang /lib/modules/4.15.7/kernel/drivers/video/nvidia.ko | head -1
++ strings -a /lib/modules/4.15.7/kernel/drivers/video/nvidia.ko
++ grep clang
clang version 5.0.0-3 (tags/RELEASE_500/final)
$ eclang /lib/modules/4.15.7/kernel/drivers/video/nvidia-drm.ko | head -1
++ strings -a /lib/modules/4.15.7/kernel/drivers/video/nvidia-drm.ko
++ grep clang
clang version 5.0.0-3 (tags/RELEASE_500/final)
$ eclang /lib/modules/4.15.7/kernel/drivers/video/nvidia-uvm.ko | head -1
++ strings -a /lib/modules/4.15.7/kernel/drivers/video/nvidia-uvm.ko
++ grep clang
clang version 5.0.0-3 (tags/RELEASE_500/final)
$ eclang /lib/modules/4.15.7/kernel/drivers/video/nvidia-modeset.ko | head -1
++ strings -a /lib/modules/4.15.7/kernel/drivers/video/nvidia-modeset.ko
++ grep clang
clang version 5.0.0-3 (tags/RELEASE_500/final)
# DMI specific hardware information - verifies and confirms the physical hardware used
# instead of QEMU or any other virtual platform for running clang-built kernel and
# Nvidia 390.25 driver.
#
$ sudo dmidecode -t bios | grep Vendor
Vendor: LENOVO
$ sudo dmidecode -t system | grep "Manufacturer:\|Version"
Manufacturer: LENOVO
Version: Lenovo Y50-70 Touch
$ sudo dmidecode -t processor | grep Version
Version: Intel(R) Core(TM) i7-4710HQ CPU @ 2.50GHz
NOTE 1: For all the detailed logs (from each stage), you can read my answer for LLVM-Clang/Linux kernel question here :Building Linux kernel with LLVM/clang.
NOTE 2: clang's warning are more pedantic than gcc as I saw. But, for the time being, to get the kernel/nvidia driver up and running, I have ignored them to see if they are not serious (so far they are not).
UPDATE 1 : For the sake of universality, I repeated the above with subsequent kernel versions (4.15.10
and 4.15.11
which is latest as of mid-March 2018) and latest Nvidia graphics GTX GeForce driver v390.42 which is latest as of mid-March 2018 - both of them built using clang
as earlier after updating kernel 4.15.11 code in IRQ work tick logic and live instruction update logic as I did above for 4.15.7, and finally installed both of them on my system hardware directly (no Qemu/kvm experiments this time as the earlier 4.15.7 kernel had come up successfully after working upon IRQ work tick logic and live instruction update logic). With that, again, Ubuntu 17.10 x86_64 successfully came up on my system hardware, on clang-built kernel 4.15.11 and clang-built latest Nvidia driver v390.42, as it did in the earlier case of kernel 4.15.7 and Nvidia driver v390.25.
Below are the details for the Ubuntu 17.10 x86_64 running on clang-built latest v4.15.11 kernel and clang-built latest Nvidia driver v390.42 scenario:
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 17.10
Release: 17.10
Codename: artful
$ cat /proc/version
Linux version 4.15.11 (exp@exp) (clang version 5.0.0-3 (tags/RELEASE_500/final)) #2 SMP Tue Mar 20 05:37:06 PDT 2018
$ eclang /lib/modules/4.15.11/build/vmlinux | grep Linux
++ strings -a /lib/modules/4.15.11/build/vmlinux
++ grep clang
Linux version 4.15.11 (exp@exp) (clang version 5.0.0-3 (tags/RELEASE_500/final)) #2 SMP Tue Mar 20 05:37:06 PDT 2018
$ sudo dmidecode -t system | grep "Manufacturer:\|Version"
Manufacturer: LENOVO
Version: Lenovo Y50-70 Touch
$ dmesg | grep NVIDIA
[ 19.491452] nvidia: module license 'NVIDIA' taints kernel.
[ 19.499360] NVRM: loading NVIDIA UNIX x86_64 Kernel Module 390.42 Sat Mar 3 04:10:22 PST 2018 (using threaded interrupts)
[ 19.542830] nvidia-modeset: Loading NVIDIA Kernel Mode Setting Driver for UNIX platforms 390.42 Sat Mar 3 03:30:48 PST 2018
$ nvidia-smi
Tue Mar 20 22:15:00 2018
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 390.42 Driver Version: 390.42 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 860M Off | 00000000:01:00.0 Off | N/A |
| N/A 52C P8 N/A / N/A | 451MiB / 4046MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
$ glxinfo | grep NVIDIA
server glx vendor string: NVIDIA Corporation
client glx vendor string: NVIDIA Corporation
OpenGL vendor string: NVIDIA Corporation
OpenGL core profile version string: 4.5.0 NVIDIA 390.42
OpenGL core profile shading language version string: 4.50 NVIDIA
OpenGL version string: 4.6.0 NVIDIA 390.42
OpenGL shading language version string: 4.60 NVIDIA
OpenGL ES profile version string: OpenGL ES 3.2 NVIDIA 390.42
$ lsmod | grep nv
nvidia_uvm 815104 0
nvidia_drm 24576 2
nvidia_modeset 1105920 5 nvidia_drm
nvidia 14368768 334 nvidia_uvm,nvidia_modeset
ipmi_msghandler 61440 2 nvidia,ipmi_devintf
drm 454656 6 nvidia_drm,i915,drm_kms_helper
$ ~/nvidia-driver-compiler.sh
++ head -1
+++ uname -r
++ eclang /lib/modules/4.15.11/kernel/drivers/video/nvidia.ko
+++ strings -a /lib/modules/4.15.11/kernel/drivers/video/nvidia.ko
+++ grep clang
clang version 5.0.0-3 (tags/RELEASE_500/final)
++ head -1
+++ uname -r
++ eclang /lib/modules/4.15.11/kernel/drivers/video/nvidia-drm.ko
+++ strings -a /lib/modules/4.15.11/kernel/drivers/video/nvidia-drm.ko
+++ grep clang
clang version 5.0.0-3 (tags/RELEASE_500/final)
+++ uname -r
++ head -1
++ eclang /lib/modules/4.15.11/kernel/drivers/video/nvidia-modeset.ko
+++ strings -a /lib/modules/4.15.11/kernel/drivers/video/nvidia-modeset.ko
+++ grep clang
clang version 5.0.0-3 (tags/RELEASE_500/final)
+++ uname -r
++ head -1
++ eclang /lib/modules/4.15.11/kernel/drivers/video/nvidia-uvm.ko
+++ strings -a /lib/modules/4.15.11/kernel/drivers/video/nvidia-uvm.ko
+++ grep clang
clang version 5.0.0-3 (tags/RELEASE_500/final)
$ ~/nvidia_driver_modules_version.sh
++ grep '^version='
+++ uname -r
++ strings -a /lib/modules/4.15.11/kernel/drivers/video/nvidia.ko
version=390.42
++ grep '^version='
+++ uname -r
++ strings -a /lib/modules/4.15.11/kernel/drivers/video/nvidia-drm.ko
version=390.42
++ grep '^version='
+++ uname -r
++ strings -a /lib/modules/4.15.11/kernel/drivers/video/nvidia-uvm.ko
+++ uname -r
++ grep '^version='
++ strings -a /lib/modules/4.15.11/kernel/drivers/video/nvidia-modeset.ko
version=390.42