5

I'm struggling to find the reason why my Ubuntu server hangs/crashes.

I have installed a barebone Ubuntu Server 20.04.2 LTS on my old computer to try and use it as a personal server to host some projects of mine.

My issue is after a few hours, sometimes as low as 2 to 3 hours or as long as 8 to 12, my server will eventually hang.

I get a visual output, but it is completely unresponsive to either keyboard, mouse or ssh. Sometimes it will log CPU #X stuck for Xs! but never seems to be coming from the same place. Sometimes kworkers, sometimes md-udev, sometimes systemd-networkd. Other times I get just no visual output at all and just have to force a reboot.

So currently running on my server are:

  • UFW
  • OpenVPN
  • Cockpit
  • Portainer
  • Docker container with nginx reverse proxy
  • Let's encrypt docker container
  • 2 Nextcloud docker containers
  • around 5 other docker containers running small apps/web apps

What's puzzling me the most is I get not logs at all, I've checked my kernel and system logs and usually the last line is an unrelated UFW block command of some spammer ip address but nothing more substantial. I've also tried looking at /var/crash/ but same thing, either empty or the log of my reboot.

I've looked around this website and people usually seem to indicate this might be a hardware issue but I've never had an issue running this system overnight on Windows 10 when it was my main setup.

What I've tried:

  • Multiple clean reinstall of Ubuntu Server 20.04.2
  • Updated my BIOS and removed any overclock I might have applied
  • Checked my temperatures and system load, both OK
  • Checked my RAM with Memtest86 for 3x4 runs, totalling over 12 hours and got no errors

My system specifications are:

Ryzen 1700 @stock
16 GB CORSAIR GDDR4 @2133MHz but rated for 3200Mhz
MSI B350 Gaming PLUS MOBO
Gigabyte GTX 1080
Samsung 960 EVO 512GB nVme SSD 
650W corsair power supply
Plenty of cooling

If any of you has any lead or advice on how to diagnose my setup, or some more steps I could try I would be grateful.

Thanks for your help


EDIT 2021-03-03

After some more digging about ubuntu ryzen and freezes, I came across this thread on AMD forums. This suggests that there are issues with the C6 state on ryzen 1st gen. Their advice is to switch the power state in BIOS to Typical Current Idle which apparently has an impact on C6 state. I'll try that and see if anything changes.

5
  • You've tagged crash, so what .crash files are there? You mention hanging (which isn't a crash situation) but have tagged crash so what did the crash files show? you seem to imply the /var/crash/ directory is empty (you do refer to it as a file? it should be a directory)
    – guiverc
    Mar 3, 2021 at 0:05
  • @guiverc I've tagged crash cause I have not found a hang tag to apply. There are no .crash files and yes as you have correctly assessed my /var/crash/ dir is empty. Sorry for the confusion
    – CreepMania
    Mar 3, 2021 at 8:56
  • 2
    If you search for "ubuntu 20.04 freeze", you'll get a LOT of results. I was having similar problems between 18.10 and 20.04 on Ryzen with a Desktop installation. I put it down to bleeding edge graphics drivers, because it felt more stable in desktop environments without compositor; but with your case, I'd check whether a different IO scheduler might help. Because: you get no logs, which means that disk IO did freeze up. Also, since you get visual output, try to run dmesg -w and leave that console open until the machine freezes again.
    – orithena
    Mar 3, 2021 at 9:54
  • @orithena Thanks I'll try that
    – CreepMania
    Mar 3, 2021 at 9:56
  • long shot (with similar hardware and 2 years of similar lockups) but openvpn with tcp might work better than udp
    – pierrely
    Mar 4, 2021 at 5:31

3 Answers 3

1

It's got to be a terribly frustrating situation no doubt. Without providing the answer a basic process for troubleshooting no doubt would be to replace each (or batches) of components (whether hardware or software.)

My digression: I have a Xigmanas BSD system which would run fine sometimes for a few days or weeks until an external array would freeze up and even trying to perform the requisite zfs resets on the filesystem weren't enough to keep the system going. Didn't seem like there should be a reason for the whole system to lock up but that was the result. Removed the array and everything has been going great for months.

Similarly I have a MacBook pro that runs under most all situations but when I had screen capture software running it would lock up so regularly that I just gave up trying to use that software on that machine (works great everywhere else!)


It does seem like a phantom hardware issue though, perhaps only exhibited under certain software configurations.

Simply put - try a different setup, if for no other reason than further problem isolation. Try a Debian release or CentOS (insert other comments on OS choices here). Is there any critical reason to use 20.04?

Changing the OS has got to be one of the easiest unless you have other compatible spare hardware (HD, memory, MB etc)

Sorry this isn't the magic "here is the answer or obvious path to the logfile with the answer" answer. :)

2
  • Not the answer I wanted, but maybe the answer I needed to hear lol I'll eventually switch OS if nothing comes up as a solution
    – CreepMania
    Mar 3, 2021 at 8:58
  • 1
    Indeed, I hope the further updates regarding Ryzen CPUs is helpful. It's crazy when we just think "everything should be working fine" and yet reality slaps us in the face :)
    – Brian Lamb
    Mar 8, 2021 at 2:41
1

I also had the same issue. I have tried many ways to solve it but nothing worked. Insead I did install other version of UBUNTU 20.04, ie UBUNTU 20.04.2.0 (FOCAL FOSSA) or Ubuntu 20.10.

2
  • 2
    Both 20.04 and 20.04.2 are Focal Fossa. The .2 release is a rollup of all the updates that have come out between April 2020 and January 2021 🤐
    – user1091774
    Mar 3, 2021 at 7:36
  • I had to go back to 18.04 as I couldn't sort out what was goofing up 20.04. Least on my ODROID XU4 I did later find that ulimit was set low and that was contributing to some of the pain. It sill randomly hangs but its not as consistent as it was on 20.04 (every 20 mins).
    – Mark
    Jul 26, 2021 at 14:45
1

Alright, I think I'll answer my own question.

My problem might have been linked to Ryzen CPUs not playing nice with Linux. According to this AMD forum tread, a lot a people have encountered similar issues with Ryzen CPUs and Linux while idle and a Reddit thread with OP reporting similar issues. There seem to be a connection between C6 CPU state and hangs.

How to fix it

BIOS

Look into your BIOS and try to find Power Supply Idle Control or Global C-state Control and change that value from Auto or Low Current Idle to Typical Current Idle.

Disable C6 state

If that didn't fix it or you didn't have the option, another way to disable it can be found at this github repo.


I personnaly had the option in my MSI motherboard BIOS, applied the correct value and been running 48h without any problems, albeit maybe with a slightly higher power consumption.

This sure has been a frustrating issue to debug, but in the end I ended up learning some new stuff and make my server somewhat stable again.

Thanks to everyone who participated and shared my frustation, I hope this thread might be useful to someone else.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .