1

I recently upgraded my Nvidia drivers from 418 to 435 (I have an RTX 2070). Long story but I had some issues with driver conflicts, so I had to switch to the nouveau drivers and back again. I'm pretty sure this has purged the 418 drivers based on dpkg output.

However, I am left with a weird issue: every once in a while, my system hangs when I try to put it into standby. I get a completely unresponsive blank screen, and I have to manually reboot, which is frustrating as I lose all my work. Strangely, this doesn't happen for every standby, and I can't figure out why.

I have tried adding 'initcall_debug' to the GRUB command line at startup, but I'm not sure if it's relevant to my situation. In any case, here's the output for lines that didn't have returned 0:

$ dmesg -s 128000 | grep "initcall" | sed "s/\(.*\)after\(.*\)/\2 \1/g" | sort -n | grep 'returned -'
 0 usecs [    0.176011] initcall xen_pvh_gnttab_setup+0x0/0x3a returned -19 
 0 usecs [    0.176400] initcall __gnttab_init+0x0/0x40 returned -19 
 0 usecs [    0.176444] initcall bts_init+0x0/0xc2 returned -19 
 0 usecs [    0.176444] initcall numachip_timer_init+0x0/0x5a returned -19 
 0 usecs [    0.176444] initcall pt_init+0x0/0x352 returned -19 
 0 usecs [    0.176444] initcall setup_vcpu_hotplug_event+0x0/0x31 returned -19 
 0 usecs [    0.176444] initcall xenbus_init+0x0/0x2d1 returned -19 
 0 usecs [    0.176444] initcall xen_pcpu_init+0x0/0xc5 returned -19 
 0 usecs [    0.276002] initcall balloon_init+0x0/0x1ba returned -19 
 0 usecs [    0.276002] initcall xen_acpi_pad_init+0x0/0x52 returned -19 
 0 usecs [    0.276002] initcall xen_setup_shutdown_event+0x0/0x40 returned -19 
 0 usecs [    0.277672] initcall save_microcode_in_initrd+0x0/0xa0 returned -22 
 0 usecs [    0.808110] initcall calgary_fixup_tce_spaces+0x0/0x106 returned -19 
 0 usecs [    0.808911] initcall amd_iommu_pc_init+0x0/0x22a returned -19 
 0 usecs [    0.808916] initcall intel_uncore_init+0x0/0x24d returned -19 
 0 usecs [    0.810834] initcall efivarfs_init+0x0/0x39 returned -19 
 0 usecs [    0.839467] initcall intel_idle_init+0x0/0x5e2 returned -19 
 0 usecs [    0.841296] initcall bgrt_init+0x0/0xba returned -19 
 0 usecs [    0.841303] initcall ghes_init+0x0/0x142 returned -19 
 0 usecs [    0.841336] initcall xenbus_probe_initcall+0x0/0x53 returned -19 
 0 usecs [    0.841338] initcall xenbus_init+0x0/0x3b returned -19 
 0 usecs [    0.841340] initcall xenbus_backend_init+0x0/0x4d returned -19 
 0 usecs [    0.841341] initcall hypervisor_subsys_init+0x0/0x2a returned -19 
 0 usecs [    0.841342] initcall hyper_sysfs_init+0x0/0x191 returned -19 
 0 usecs [    0.841349] initcall xen_late_init_mcelog+0x0/0x66 returned -19 
 0 usecs [    0.841351] initcall xen_acpi_processor_init+0x0/0x1d9 returned -19 
 0 usecs [    0.841381] initcall xen_hvc_init+0x0/0x24b returned -19 
 0 usecs [    0.865921] initcall xlblk_init+0x0/0xf0 returned -19 
 0 usecs [    0.866405] initcall netif_init+0x0/0x77 returned -19 
 0 usecs [    1.187288] initcall intel_pstate_init+0x0/0xb3 returned -19 
 0 usecs [    1.191312] initcall efivars_sysfs_init+0x0/0x210 returned -19 
 0 usecs [    1.191314] initcall esrt_sysfs_init+0x0/0x2d9 returned -38 
 0 usecs [    1.191522] initcall pmc_core_probe+0x0/0x242 returned -19 
 0 usecs [    1.317488] initcall boot_wait_for_devices+0x0/0x2e returned -19 
 0 usecs [    1.330870] initcall powernowk8_init+0x0/0x1e0 returned -19 
 0 usecs [    1.330873] initcall centrino_init+0x0/0x30 returned -19 
 0 usecs [    1.336981] initcall efi_shutdown_init+0x0/0x44 returned -19 
 0 usecs [    1.336983] initcall itmt_legacy_init+0x0/0x5b returned -19 
 0 usecs [    1.336989] initcall software_resume+0x0/0x2a0 returned -2 
 4 usecs [    1.191529] initcall pmc_atom_init+0x0/0x2b2 returned -19 
 7 usecs [    0.865931] initcall htcpld_core_init+0x0/0x32 returned -19 
 10 usecs [    9.639314] initcall amd64_edac_init+0x0/0x1000 [amd64_edac_mod] returned -19 
 10 usecs [    9.947048] initcall amd64_edac_init+0x0/0x1000 [amd64_edac_mod] returned -19 
 12 usecs [    8.432809] initcall amd64_edac_init+0x0/0x1000 [amd64_edac_mod] returned -19 
 12 usecs [    9.328536] initcall amd64_edac_init+0x0/0x1000 [amd64_edac_mod] returned -19 
 13 usecs [    7.579790] initcall amd64_edac_init+0x0/0x1000 [amd64_edac_mod] returned -19 
 15 usecs [    7.760494] initcall amd64_edac_init+0x0/0x1000 [amd64_edac_mod] returned -19 
 15 usecs [    8.032654] initcall amd64_edac_init+0x0/0x1000 [amd64_edac_mod] returned -19 
 16 usecs [    9.200728] initcall amd64_edac_init+0x0/0x1000 [amd64_edac_mod] returned -19 
 17 usecs [    7.652451] initcall amd64_edac_init+0x0/0x1000 [amd64_edac_mod] returned -19 
 17 usecs [    9.421129] initcall amd64_edac_init+0x0/0x1000 [amd64_edac_mod] returned -19 
 18 usecs [    7.932705] initcall amd64_edac_init+0x0/0x1000 [amd64_edac_mod] returned -19 
 20 usecs [    8.932645] initcall amd64_edac_init+0x0/0x1000 [amd64_edac_mod] returned -19 
 22 usecs [    0.863365] initcall agp_amd64_mod_init+0x0/0x26 returned -19 
 57 usecs [    2.464629] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 57 usecs [    2.568959] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 57 usecs [    2.656072] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 57 usecs [    2.920610] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 58 usecs [    2.376457] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 58 usecs [    2.420686] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 58 usecs [    2.850913] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 65 usecs [    2.728445] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 65 usecs [    2.800692] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 66 usecs [    2.116499] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 66 usecs [    2.316528] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 66 usecs [    2.496452] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 66 usecs [    2.768653] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 72 usecs [    2.888681] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 76 usecs [    2.220657] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 78 usecs [    2.612693] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 80 usecs [    2.688437] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 89 usecs [    9.421155] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 91 usecs [    2.276836] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 95 usecs [    2.072491] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 97 usecs [    1.955728] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 104 usecs [    2.176572] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 108 usecs [   10.176986] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 109 usecs [   10.080899] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 109 usecs [   10.216712] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 109 usecs [   10.313008] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 109 usecs [   10.373069] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 115 usecs [   10.042520] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 120 usecs [   10.261266] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 124 usecs [    7.652694] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 129 usecs [    7.760871] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 138 usecs [    8.433073] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 138 usecs [    9.201083] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 141 usecs [    7.403532] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 147 usecs [    7.561564] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 152 usecs [    7.933158] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 155 usecs [    7.489113] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 158 usecs [   10.124370] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 165 usecs [    8.032815] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 167 usecs [    8.932793] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 179 usecs [    9.946483] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 183 usecs [    9.639479] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 249 usecs [    9.329127] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 270 usecs [    1.851350] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 3545 usecs [    1.161088] initcall i8042_init+0x0/0x474 returned -19 
 5940 usecs [    1.336958] initcall edd_init+0x0/0x2d8 returned -19 

Anybody have any better ways to debug standby issues?

Additional question: I'm kinda hesitant to perform a full purge of the Nvidia drivers again as I have a working Tensorflow configuration that I don't want to mess with. Does purging mean I will have to reinstall the CUDA drivers? If so, can this be done independently of other packages necessary for Tensorflow?

Thanks a lot!

EDIT: I tried going into standby using sudo sh -c "sync && echo 1 > /sys/power/pm_trace && pm-suspend" and checking dmesg output afterwards. I found the following output:

[    1.320847]   Magic number: 0:236:250
[    1.323713]   hash matches /build/linux-UDHJtG/linux-4.15.0/drivers/base/power/main.c:1559
[    1.326768] pci 0000:0a:00.0: hash matches
[    1.329815] memory memory124: hash matches

After checking lspci I found that 0000:0a:00.0 is 'VGA compatible controller: NVIDIA Corporation Device 1f07 (rev a1)' so looks like I'm gonna have to try purging the Nvidia drivers again. However, I have no idea what memory memory 124 is - anybody know?

EDIT 2: Purged and reinstalled Nvidia drivers completely. Same problem is happening. However, I've noticed something: it only occurs when I have Spyder open via Anaconda and have started training neural net models using my GPU. Any ideas why?

EDIT 3: Went back to Nvidia 418 drivers. Same problem occurring, so I switched back to 435 drivers. Here's my output from dpkg -l | grep -i nvidia:

ii  cuda-nsight-compute-10-1                           10.1.243-1                                       amd64        NVIDIA Nsight Compute
ii  cuda-nsight-systems-10-1                           10.1.243-1                                       amd64        NVIDIA Nsight Systems
ii  cuda-nvtx-10-1                                     10.1.243-1                                       amd64        NVIDIA Tools Extension
ii  libnvidia-cfg1-435:amd64                           435.21-0ubuntu0.18.04.2                          amd64        NVIDIA binary OpenGL/GLX configuration library
ii  libnvidia-common-435                               435.21-0ubuntu0.18.04.2                          all          Shared files used by the NVIDIA libraries
ii  libnvidia-compute-435:amd64                        435.21-0ubuntu0.18.04.2                          amd64        NVIDIA libcompute package
ii  libnvidia-compute-435:i386                         435.21-0ubuntu0.18.04.2                          i386         NVIDIA libcompute package
ii  libnvidia-decode-435:amd64                         435.21-0ubuntu0.18.04.2                          amd64        NVIDIA Video Decoding runtime libraries
ii  libnvidia-decode-435:i386                          435.21-0ubuntu0.18.04.2                          i386         NVIDIA Video Decoding runtime libraries
ii  libnvidia-encode-435:amd64                         435.21-0ubuntu0.18.04.2                          amd64        NVENC Video Encoding runtime library
ii  libnvidia-encode-435:i386                          435.21-0ubuntu0.18.04.2                          i386         NVENC Video Encoding runtime library
ii  libnvidia-fbc1-435:amd64                           435.21-0ubuntu0.18.04.2                          amd64        NVIDIA OpenGL-based Framebuffer Capture runtime library
ii  libnvidia-fbc1-435:i386                            435.21-0ubuntu0.18.04.2                          i386         NVIDIA OpenGL-based Framebuffer Capture runtime library
ii  libnvidia-gl-435:amd64                             435.21-0ubuntu0.18.04.2                          amd64        NVIDIA OpenGL/GLX/EGL/GLES GLVND libraries and Vulkan ICD
ii  libnvidia-gl-435:i386                              435.21-0ubuntu0.18.04.2                          i386         NVIDIA OpenGL/GLX/EGL/GLES GLVND libraries and Vulkan ICD
ii  libnvidia-ifr1-435:amd64                           435.21-0ubuntu0.18.04.2                          amd64        NVIDIA OpenGL-based Inband Frame Readback runtime library
ii  libnvidia-ifr1-435:i386                            435.21-0ubuntu0.18.04.2                          i386         NVIDIA OpenGL-based Inband Frame Readback runtime library
ii  nsight-compute-2019.4.0                            2019.4.0.12-1                                    amd64        NVIDIA Nsight Compute
ii  nsight-systems-2019.3.7                            2019.3.7.5-3837e03                               amd64        NVIDIA Nsight Systems is a statistical sampling profiler with tracing features.
ii  nvidia-compute-utils-435                           435.21-0ubuntu0.18.04.2                          amd64        NVIDIA compute utilities
ii  nvidia-dkms-435                                    435.21-0ubuntu0.18.04.2                          amd64        NVIDIA DKMS package
ii  nvidia-driver-435                                  435.21-0ubuntu0.18.04.2                          amd64        NVIDIA driver metapackage
ii  nvidia-kernel-common-435                           435.21-0ubuntu0.18.04.2                          amd64        Shared files used with the kernel module
ii  nvidia-kernel-source-435                           435.21-0ubuntu0.18.04.2                          amd64        NVIDIA kernel source package
ii  nvidia-prime                                       0.8.8.2                                          all          Tools to enable NVIDIA's Prime
ii  nvidia-settings                                    418.87.00-0ubuntu1                               amd64        Tool for configuring the NVIDIA graphics driver
ii  nvidia-utils-435                                   435.21-0ubuntu0.18.04.2                          amd64        NVIDIA driver support binaries
ii  xserver-xorg-video-nvidia-435                      435.21-0ubuntu0.18.04.2                          amd64        NVIDIA binary Xorg driver

4
  • 1
    Your concerns about CUDA dependencies are valid, some CUDA installation methods tie things to their supplied 418 Nvidia drivers. However, your problem is more likely memory/swap related -- what does free report for swap? please post dpkg-l |grep -i nvidia also to see the state of the Nvidia drivers. – ubfan1 Apr 12 '20 at 21:42
  • Swap reports 0 used, as I have a low swappiness setting. Plenty of free RAM and VRAM - the models I'm running aren't complicated. I've included the output from dpkg above. – Piethon Apr 12 '20 at 22:31
  • 1
    Nvidia drivers look OK. Did you make/run all the CUDA samples and DNN samples? Sometimes you can pick up a missing library that way. At least one of the CUDA sample compliations goes hard into swapping on my 12GB machine. Just installed a Tensorflow setup like yours on U20.04Beta (with the 440 drivers), but haven't done much with it yet.Maybe try a HWE package to get to the 5.3 (?) kernel and the 440 Nvidia drivers? – ubfan1 Apr 12 '20 at 23:11
  • 1
    Turns out it probably is a memory issue - I just checked nvidia-smi and found that when I run a Tensorflow model it uses virtually all my GPU memory by default (7725/7977 MiB being reported as used). tensorflow.org/guide/gpu suggests this is standard Tensorflow practice. As soon as I close Spyder the memory usage drops back down again to around 500 MiB, and I can put the system in standby again. But this is still quite annoying - I'd like to be able to leave Spyder open. – Piethon Apr 12 '20 at 23:26
2

Found a workaround - I have to set memory growth to TRUE as detailed in https://www.tensorflow.org/guide/gpu#allowing_gpu_memory_growth. Slightly annoying but necessary I suppose. Thanks @ubfan1 for your help!

If anybody knows how to fix this issue completely, I would be very grateful.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.