1

I recently upgraded my Nvidia drivers from 418 to 435 (I have an RTX 2070). Long story but I had some issues with driver conflicts, so I had to switch to the nouveau drivers and back again. I'm pretty sure this has purged the 418 drivers based on dpkg output.

However, I am left with a weird issue: every once in a while, my system hangs when I try to put it into standby. I get a completely unresponsive blank screen, and I have to manually reboot, which is frustrating as I lose all my work. Strangely, this doesn't happen for every standby, and I can't figure out why.

I have tried adding 'initcall_debug' to the GRUB command line at startup, but I'm not sure if it's relevant to my situation. In any case, here's the output for lines that didn't have returned 0:

$ dmesg -s 128000 | grep "initcall" | sed "s/\(.*\)after\(.*\)/\2 \1/g" | sort -n | grep 'returned -'
 0 usecs [    0.176011] initcall xen_pvh_gnttab_setup+0x0/0x3a returned -19 
 0 usecs [    0.176400] initcall __gnttab_init+0x0/0x40 returned -19 
 0 usecs [    0.176444] initcall bts_init+0x0/0xc2 returned -19 
 0 usecs [    0.176444] initcall numachip_timer_init+0x0/0x5a returned -19 
 0 usecs [    0.176444] initcall pt_init+0x0/0x352 returned -19 
 0 usecs [    0.176444] initcall setup_vcpu_hotplug_event+0x0/0x31 returned -19 
 0 usecs [    0.176444] initcall xenbus_init+0x0/0x2d1 returned -19 
 0 usecs [    0.176444] initcall xen_pcpu_init+0x0/0xc5 returned -19 
 0 usecs [    0.276002] initcall balloon_init+0x0/0x1ba returned -19 
 0 usecs [    0.276002] initcall xen_acpi_pad_init+0x0/0x52 returned -19 
 0 usecs [    0.276002] initcall xen_setup_shutdown_event+0x0/0x40 returned -19 
 0 usecs [    0.277672] initcall save_microcode_in_initrd+0x0/0xa0 returned -22 
 0 usecs [    0.808110] initcall calgary_fixup_tce_spaces+0x0/0x106 returned -19 
 0 usecs [    0.808911] initcall amd_iommu_pc_init+0x0/0x22a returned -19 
 0 usecs [    0.808916] initcall intel_uncore_init+0x0/0x24d returned -19 
 0 usecs [    0.810834] initcall efivarfs_init+0x0/0x39 returned -19 
 0 usecs [    0.839467] initcall intel_idle_init+0x0/0x5e2 returned -19 
 0 usecs [    0.841296] initcall bgrt_init+0x0/0xba returned -19 
 0 usecs [    0.841303] initcall ghes_init+0x0/0x142 returned -19 
 0 usecs [    0.841336] initcall xenbus_probe_initcall+0x0/0x53 returned -19 
 0 usecs [    0.841338] initcall xenbus_init+0x0/0x3b returned -19 
 0 usecs [    0.841340] initcall xenbus_backend_init+0x0/0x4d returned -19 
 0 usecs [    0.841341] initcall hypervisor_subsys_init+0x0/0x2a returned -19 
 0 usecs [    0.841342] initcall hyper_sysfs_init+0x0/0x191 returned -19 
 0 usecs [    0.841349] initcall xen_late_init_mcelog+0x0/0x66 returned -19 
 0 usecs [    0.841351] initcall xen_acpi_processor_init+0x0/0x1d9 returned -19 
 0 usecs [    0.841381] initcall xen_hvc_init+0x0/0x24b returned -19 
 0 usecs [    0.865921] initcall xlblk_init+0x0/0xf0 returned -19 
 0 usecs [    0.866405] initcall netif_init+0x0/0x77 returned -19 
 0 usecs [    1.187288] initcall intel_pstate_init+0x0/0xb3 returned -19 
 0 usecs [    1.191312] initcall efivars_sysfs_init+0x0/0x210 returned -19 
 0 usecs [    1.191314] initcall esrt_sysfs_init+0x0/0x2d9 returned -38 
 0 usecs [    1.191522] initcall pmc_core_probe+0x0/0x242 returned -19 
 0 usecs [    1.317488] initcall boot_wait_for_devices+0x0/0x2e returned -19 
 0 usecs [    1.330870] initcall powernowk8_init+0x0/0x1e0 returned -19 
 0 usecs [    1.330873] initcall centrino_init+0x0/0x30 returned -19 
 0 usecs [    1.336981] initcall efi_shutdown_init+0x0/0x44 returned -19 
 0 usecs [    1.336983] initcall itmt_legacy_init+0x0/0x5b returned -19 
 0 usecs [    1.336989] initcall software_resume+0x0/0x2a0 returned -2 
 4 usecs [    1.191529] initcall pmc_atom_init+0x0/0x2b2 returned -19 
 7 usecs [    0.865931] initcall htcpld_core_init+0x0/0x32 returned -19 
 10 usecs [    9.639314] initcall amd64_edac_init+0x0/0x1000 [amd64_edac_mod] returned -19 
 10 usecs [    9.947048] initcall amd64_edac_init+0x0/0x1000 [amd64_edac_mod] returned -19 
 12 usecs [    8.432809] initcall amd64_edac_init+0x0/0x1000 [amd64_edac_mod] returned -19 
 12 usecs [    9.328536] initcall amd64_edac_init+0x0/0x1000 [amd64_edac_mod] returned -19 
 13 usecs [    7.579790] initcall amd64_edac_init+0x0/0x1000 [amd64_edac_mod] returned -19 
 15 usecs [    7.760494] initcall amd64_edac_init+0x0/0x1000 [amd64_edac_mod] returned -19 
 15 usecs [    8.032654] initcall amd64_edac_init+0x0/0x1000 [amd64_edac_mod] returned -19 
 16 usecs [    9.200728] initcall amd64_edac_init+0x0/0x1000 [amd64_edac_mod] returned -19 
 17 usecs [    7.652451] initcall amd64_edac_init+0x0/0x1000 [amd64_edac_mod] returned -19 
 17 usecs [    9.421129] initcall amd64_edac_init+0x0/0x1000 [amd64_edac_mod] returned -19 
 18 usecs [    7.932705] initcall amd64_edac_init+0x0/0x1000 [amd64_edac_mod] returned -19 
 20 usecs [    8.932645] initcall amd64_edac_init+0x0/0x1000 [amd64_edac_mod] returned -19 
 22 usecs [    0.863365] initcall agp_amd64_mod_init+0x0/0x26 returned -19 
 57 usecs [    2.464629] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 57 usecs [    2.568959] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 57 usecs [    2.656072] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 57 usecs [    2.920610] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 58 usecs [    2.376457] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 58 usecs [    2.420686] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 58 usecs [    2.850913] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 65 usecs [    2.728445] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 65 usecs [    2.800692] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 66 usecs [    2.116499] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 66 usecs [    2.316528] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 66 usecs [    2.496452] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 66 usecs [    2.768653] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 72 usecs [    2.888681] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 76 usecs [    2.220657] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 78 usecs [    2.612693] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 80 usecs [    2.688437] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 89 usecs [    9.421155] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 91 usecs [    2.276836] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 95 usecs [    2.072491] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 97 usecs [    1.955728] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 104 usecs [    2.176572] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 108 usecs [   10.176986] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 109 usecs [   10.080899] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 109 usecs [   10.216712] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 109 usecs [   10.313008] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 109 usecs [   10.373069] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 115 usecs [   10.042520] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 120 usecs [   10.261266] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 124 usecs [    7.652694] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 129 usecs [    7.760871] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 138 usecs [    8.433073] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 138 usecs [    9.201083] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 141 usecs [    7.403532] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 147 usecs [    7.561564] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 152 usecs [    7.933158] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 155 usecs [    7.489113] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 158 usecs [   10.124370] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 165 usecs [    8.032815] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 167 usecs [    8.932793] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 179 usecs [    9.946483] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 183 usecs [    9.639479] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 249 usecs [    9.329127] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 270 usecs [    1.851350] initcall fjes_init_module+0x0/0xd0 [fjes] returned -19 
 3545 usecs [    1.161088] initcall i8042_init+0x0/0x474 returned -19 
 5940 usecs [    1.336958] initcall edd_init+0x0/0x2d8 returned -19 

Anybody have any better ways to debug standby issues?

Additional question: I'm kinda hesitant to perform a full purge of the Nvidia drivers again as I have a working Tensorflow configuration that I don't want to mess with. Does purging mean I will have to reinstall the CUDA drivers? If so, can this be done independently of other packages necessary for Tensorflow?

Thanks a lot!

EDIT: I tried going into standby using sudo sh -c "sync && echo 1 > /sys/power/pm_trace && pm-suspend" and checking dmesg output afterwards. I found the following output:

[    1.320847]   Magic number: 0:236:250
[    1.323713]   hash matches /build/linux-UDHJtG/linux-4.15.0/drivers/base/power/main.c:1559
[    1.326768] pci 0000:0a:00.0: hash matches
[    1.329815] memory memory124: hash matches

After checking lspci I found that 0000:0a:00.0 is 'VGA compatible controller: NVIDIA Corporation Device 1f07 (rev a1)' so looks like I'm gonna have to try purging the Nvidia drivers again. However, I have no idea what memory memory 124 is - anybody know?

EDIT 2: Purged and reinstalled Nvidia drivers completely. Same problem is happening. However, I've noticed something: it only occurs when I have Spyder open via Anaconda and have started training neural net models using my GPU. Any ideas why?

EDIT 3: Went back to Nvidia 418 drivers. Same problem occurring, so I switched back to 435 drivers. Here's my output from dpkg -l | grep -i nvidia:

ii  cuda-nsight-compute-10-1                           10.1.243-1                                       amd64        NVIDIA Nsight Compute
ii  cuda-nsight-systems-10-1                           10.1.243-1                                       amd64        NVIDIA Nsight Systems
ii  cuda-nvtx-10-1                                     10.1.243-1                                       amd64        NVIDIA Tools Extension
ii  libnvidia-cfg1-435:amd64                           435.21-0ubuntu0.18.04.2                          amd64        NVIDIA binary OpenGL/GLX configuration library
ii  libnvidia-common-435                               435.21-0ubuntu0.18.04.2                          all          Shared files used by the NVIDIA libraries
ii  libnvidia-compute-435:amd64                        435.21-0ubuntu0.18.04.2                          amd64        NVIDIA libcompute package
ii  libnvidia-compute-435:i386                         435.21-0ubuntu0.18.04.2                          i386         NVIDIA libcompute package
ii  libnvidia-decode-435:amd64                         435.21-0ubuntu0.18.04.2                          amd64        NVIDIA Video Decoding runtime libraries
ii  libnvidia-decode-435:i386                          435.21-0ubuntu0.18.04.2                          i386         NVIDIA Video Decoding runtime libraries
ii  libnvidia-encode-435:amd64                         435.21-0ubuntu0.18.04.2                          amd64        NVENC Video Encoding runtime library
ii  libnvidia-encode-435:i386                          435.21-0ubuntu0.18.04.2                          i386         NVENC Video Encoding runtime library
ii  libnvidia-fbc1-435:amd64                           435.21-0ubuntu0.18.04.2                          amd64        NVIDIA OpenGL-based Framebuffer Capture runtime library
ii  libnvidia-fbc1-435:i386                            435.21-0ubuntu0.18.04.2                          i386         NVIDIA OpenGL-based Framebuffer Capture runtime library
ii  libnvidia-gl-435:amd64                             435.21-0ubuntu0.18.04.2                          amd64        NVIDIA OpenGL/GLX/EGL/GLES GLVND libraries and Vulkan ICD
ii  libnvidia-gl-435:i386                              435.21-0ubuntu0.18.04.2                          i386         NVIDIA OpenGL/GLX/EGL/GLES GLVND libraries and Vulkan ICD
ii  libnvidia-ifr1-435:amd64                           435.21-0ubuntu0.18.04.2                          amd64        NVIDIA OpenGL-based Inband Frame Readback runtime library
ii  libnvidia-ifr1-435:i386                            435.21-0ubuntu0.18.04.2                          i386         NVIDIA OpenGL-based Inband Frame Readback runtime library
ii  nsight-compute-2019.4.0                            2019.4.0.12-1                                    amd64        NVIDIA Nsight Compute
ii  nsight-systems-2019.3.7                            2019.3.7.5-3837e03                               amd64        NVIDIA Nsight Systems is a statistical sampling profiler with tracing features.
ii  nvidia-compute-utils-435                           435.21-0ubuntu0.18.04.2                          amd64        NVIDIA compute utilities
ii  nvidia-dkms-435                                    435.21-0ubuntu0.18.04.2                          amd64        NVIDIA DKMS package
ii  nvidia-driver-435                                  435.21-0ubuntu0.18.04.2                          amd64        NVIDIA driver metapackage
ii  nvidia-kernel-common-435                           435.21-0ubuntu0.18.04.2                          amd64        Shared files used with the kernel module
ii  nvidia-kernel-source-435                           435.21-0ubuntu0.18.04.2                          amd64        NVIDIA kernel source package
ii  nvidia-prime                                       0.8.8.2                                          all          Tools to enable NVIDIA's Prime
ii  nvidia-settings                                    418.87.00-0ubuntu1                               amd64        Tool for configuring the NVIDIA graphics driver
ii  nvidia-utils-435                                   435.21-0ubuntu0.18.04.2                          amd64        NVIDIA driver support binaries
ii  xserver-xorg-video-nvidia-435                      435.21-0ubuntu0.18.04.2                          amd64        NVIDIA binary Xorg driver

4
  • 1
    Your concerns about CUDA dependencies are valid, some CUDA installation methods tie things to their supplied 418 Nvidia drivers. However, your problem is more likely memory/swap related -- what does free report for swap? please post dpkg-l |grep -i nvidia also to see the state of the Nvidia drivers.
    – ubfan1
    Apr 12, 2020 at 21:42
  • Swap reports 0 used, as I have a low swappiness setting. Plenty of free RAM and VRAM - the models I'm running aren't complicated. I've included the output from dpkg above.
    – Piethon
    Apr 12, 2020 at 22:31
  • 1
    Nvidia drivers look OK. Did you make/run all the CUDA samples and DNN samples? Sometimes you can pick up a missing library that way. At least one of the CUDA sample compliations goes hard into swapping on my 12GB machine. Just installed a Tensorflow setup like yours on U20.04Beta (with the 440 drivers), but haven't done much with it yet.Maybe try a HWE package to get to the 5.3 (?) kernel and the 440 Nvidia drivers?
    – ubfan1
    Apr 12, 2020 at 23:11
  • 1
    Turns out it probably is a memory issue - I just checked nvidia-smi and found that when I run a Tensorflow model it uses virtually all my GPU memory by default (7725/7977 MiB being reported as used). tensorflow.org/guide/gpu suggests this is standard Tensorflow practice. As soon as I close Spyder the memory usage drops back down again to around 500 MiB, and I can put the system in standby again. But this is still quite annoying - I'd like to be able to leave Spyder open.
    – Piethon
    Apr 12, 2020 at 23:26

1 Answer 1

2

Found a workaround - I have to set memory growth to TRUE as detailed in https://www.tensorflow.org/guide/gpu#allowing_gpu_memory_growth. Slightly annoying but necessary I suppose. Thanks @ubfan1 for your help!

If anybody knows how to fix this issue completely, I would be very grateful.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .