2

I have a remote server (on the other side of the country) that has become corrupted in some way. I still have ssh access to it. Is it possible to overwrite the OS partition while it's running?

In the past I've managed to image a new SSD by using dd over ssh, but that computer was booted from a gparted usb stick. The only OS I have access to on that remote server is the one on the hard disk; no removable media is or will be available.

What will overwriting a currently running OS do? Can I load the disk image into a ram drive and run dd on the current disk? Perhaps I could make a second partition and dual boot with gparted or some other tool.

The image I will be writing to the disk has ssh with default credentials already enabled, so once I boot into the new image I should be able gain access.

This is my own hardware running at a remote location. The only access I have is through SSH

Improve this question
14
  • what is "corrupted"? have you tried a file system repair first? or possiblly the software that is corrupted. or.... not much to go on .......the second partition idea seems like something I, myself would go with though, or some variation of that
    – WU-TANG
    Dec 2, 2020 at 23:58
  • There are some dependency issues, missing / corrupt files after an update that are beyond my ability to remedy. My usual go to when my system gets messed up is to do a fresh install. For now my only access is SSH. I also see this as an interesting challenge.
    – Bacon Snot
    Dec 3, 2020 at 0:20
  • I would not change too much on the original system at first... The problem is, when you reboot, if for whatever reason, the system doesnt come back up, youre locked out. If you are able to make a new partition and dd your new image to it.. then I would use the original system's update-grub to get a new grub menu entry... rather than reinstalling grub (from within your new system) and risking boot/efi problems... Once youve established you can reliable ssh into your new image/partition.. then you can dd another pre-made install for the original partition, but i dont think you can "install".
    – WU-TANG
    Dec 3, 2020 at 0:30
  • Also the new pre-made install, you may want to consider installing kvm and putting your system in a VM... This would be WAAAAAAYYYYY easier if you had a corrupt VM that needed a remedy... and their are plenty of "remote" tools, simply because everything is remote by nature
    – WU-TANG
    Dec 3, 2020 at 0:35
  • Also... the risk comes in when you have to change the default system to boot into in /etc/default/grub (which I always get wrong on first try!!!)... if designate the wrong one you could end up in memtest or something and be locked out... You never stated, do you have someone on the other end that, if need be, could make the selection????
    – WU-TANG
    Dec 3, 2020 at 0:44

2 Answers 2

Reset to default
3

Here is how I got it to work:

Step 1: Create a new partition. You can use any number of utilities to do this, I used cfdisk since it was available

Step 2: Image that new partition with a live distro that has SSH enabled and default username and password. In my case the old OS was on /dev/sda1 and the new one was on /dev/sda2. The new OS must be able to accept an SSH connection from boot without any prompts, or you will lose contact with your remote computer. It must also have the network properly configured. Either set a proper static IP, subnet mask, gateway, etc; or use DHCP and use your remote router to find the computer's new IP address.

Step 3: Run update-grub to add it to the boot options.

Step 4: View the file /boot/grub/grub.cfg and find the menuentry for the new OS image. It should look something like this: menuentry "KUbuntu 16.04 amd64 desktop (live)" --class windows --class os. What you need is the label in the quotes, in this case: KUbuntu 16.04 amd64 destop (live)

Step 5: Open /etc/default/grub in any editor and look for GRUB_DEFAULT=. Make sure it is set to GRUB_DEFAULT=saved.

Step 6: Run grub-reboot followed by your chosen menu entry in quotes. In this case grub-reboot "KUbuntu 16.04 amd64 desktop (live)"

Step 7: Run sudo reboot to get your remote computer to reboot. If you have done everything right you should be able to SSH into the new system once it boots.

Step 8: From your new operating system, overwrite the image of the original OS. In my case it was on /dev/sda1. This will also overwrite the grub settings on the boot partition causing you to boot into the updated OS on the original partition next time you boot. Again, what ever OS you put on that partition must have SSH enabled, default username and password, and network settings configured already.

Step 9: Reboot and you should be in your newly imaged operating system.

Improve this answer
6
  • congrats...very nice... i will bookmark this so i can have a checklist. i'd forgot all about network, could pose a real problem without dhcp. A question/remark or two though... Step 3 did you actually have to run os_prober manually? os_prober gets ran when you run update-grub (it's one of the scripts in /etc/grub.d/). Step 4 says "Edit" but I am assuming you mean just view it? If you edit it, it gets overwritten when you run update-grub. I'm not quite sure "it will boot into the last known option", more like it will boot into the "saved entry" based on how youre currently configured
    – WU-TANG
    Dec 6, 2020 at 7:04
  • 1
    Step 5 I feel ridiculous not knowing all these years, you could use the OS name vice the enumeration. After looking into that, even more ridiculous, I found there is a much more safer approach. GRUB_DEFAULT=saved, and on the command line grub-set-default with grub-reboot can be used to set a 1-time boot selection. And even tabbing through those 2 commands even presents the OS options. I was able to boot into an older kernel 1-time with sudo grub-reboot "Advanced options for Ubuntu>Ubuntu, with Linux 5.4.0-54-generic" But, glad this worked out, I know this would have been hard to fix.
    – WU-TANG
    Dec 6, 2020 at 7:10
  • I wish I knew about grub-reboot years ago. I updated the steps to reflect your suggestions. I think it's much easier and safer using 'grub-reboot'. I think I had to run os-prober because of my broken system specifically. Probably not necessary for everyone else.
    – Bacon Snot
    Dec 7, 2020 at 14:20
  • While this is a great answer, how did you create the sda2 partition? Did you already have unpartitioned space available? Because as far as I know you cannot resize a partition while it is in use.
    – Terrance
    Dec 12, 2020 at 0:32
  • 1
    You are correct, I did have unpartitioned space available. If you are in the same situation but have no unpartitioned space available, my first idea would be to turn swap off and use the swap partition. There are a few more ideas on how to shrink and create partitions here askubuntu.com/questions/390769/….
    – Bacon Snot
    Dec 14, 2020 at 14:23
0

It is impossible to overwrite your system's installation while running that system.

You need to either have physical access to the server or access to tools provided by your server's host to do the tasks you want to do.

Consult with the hosting provider, their documentation, or the owner of the physical server to find out the method to complete your tasks.

Improve this answer
2
  • I should have mentioned, this is my own hardware running at a remote location.
    – Bacon Snot
    Dec 2, 2020 at 23:54
  • So you will need physical access or someone to do it for you.
    – Nmath
    Dec 3, 2020 at 0:43

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .