46

So I was running apt-get upgrade on a server when the router decided it had been too long since it last made me angry: It dropped all connection. Moral of the story is to use screen a lot when you're on a bum router.

Anyway, I logged back in and found in htop that the process was still hanging there, still waiting for my Y/n to upgrade (hadn't hit it yet, luckily). Is there any way I can reattach to a session that had been broken off? I ended up just killing it since it wasn't in the middle of package management but it would be great to know for future reference.

2
  • 1
    I'm surprised the apt-get process was still running. It should have died along with the whole process chain up to SSH. I've noticed that do-dist-upgrade automatically starts in a screen/byobu session: maybe in some circumstances, apt-get does the same?
    – nfirvine
    Oct 2, 2012 at 17:41
  • I had an apt upgrade running waiting for input while my ssh session broke. After reconnecting I killed the running apt process and then continued from last upgrade step with dpkg --configure -a - which should complete all pending packages. Feb 11 at 17:10

4 Answers 4

33

While you can't reattach to a broken SSH session, you can reparent the process running inside SSH – functionally equivalent to what you want.

Instructions

In your case, you would take over the apt-get process, with the goal to control it from a new SSH session, screen session or the like. My favourite for this is the reptyr command:

$ sudo apt-get install reptyr
$ ps ax | grep apt-get
10626 pts/8   R+     0:32 apt-get upgrade

Then, with the pid you found for your process:

$ sudo reptyr -T 10626

Or if that does not work, try:

$ reptyr 10626

After this stage, all your keyboard input goes to the program you took over. Unfortunately you will not see old output of the SSH session, such as the apt-get output asking you for confirmation.

Explanations

There are multiple other tools that basically work the same as reptyr (that is, via ptrace debug attachment). See the following questions and answers where they are discussed:

In the instructions above, the reptyr 10626 uses ptrace debug attachment while the sudo reptyr -T 10626 command uses TTY stealing and is preferable (details).

Finally, the reason why you can't take over a SSH session this way is because a sshd process is not controlled by a host terminal, instead it provides the slave part of a terminal – a pts device – while the master part controlling it resides on the client machine, here with a broken-down SSH session in between. When you force taking over such a sshd process with reptyr -s <pid>, your keyboard input goes to that process, not its active child process. So a "Ctrl+Z" will simply kill off that sshd.

4
  • 3
    Can you explain how you can install reptyr using apt-get in order to connect to an apt-get process that is still running? Doesn't the running process lock dpkg to prevent reptyr from being installed?
    – Mike Allen
    Aug 27, 2020 at 19:29
  • 1
    @MikeAllen Good point. I believe I used it to attach to a process that was not apt-get but then wrote the answer for this specific question without noticing the point you raised.
    – tanius
    Aug 29, 2020 at 11:36
  • 1
    Note that if you attach your process this way and you get disconnected AGAIN, then you have to find the still running reptyr PID to re-attach, not the original PID. If you try to attach to original PID while reptyr is still running you'll get "Unable to find the fd for the pty!" message.
    – pkExec
    Nov 29, 2022 at 11:26
  • 1
    @MikeAllen if you don't have reptyr installed and you're trying to recover from a situation where dpkg is locked (thereby preventing you from running apt-get or aptitude) you can upload (via sftp, scp, etc.) /usr/bin/reptyr from a compatible system and then use that binary the way @tanius describes. I.E. if you upload it to your home directory, run it via sudo ./reptyr -T .... (Afterwards you can install reptyr normally if you'd like.)
    – Sam
    Sep 23, 2023 at 19:11
23
+50

The answer to your proper question is: you can't. I think the main problem is that the authentication procedures will be out of sync. It Just Doesn't Work Like That.

As you have yourself noticed, the solution is to use screen when possible (by the way, tmux is an alternative to screen).

4
  • 2
    But what if you have passwordless ssh? Can you do it then? Aug 3, 2013 at 17:17
  • 2
    byobu is a nice, easier to use front end to screen (or tmux) - definitely worth a look (:
    – drevicko
    Feb 15, 2015 at 22:35
  • Sridhar-Samobol, authentication still needs to take place. Attaching to a running session has no way to do the initial handshake again, so it's invariants would be broken if we introduced a new session into an existing one. Answer: no.
    – kevr
    Apr 17, 2019 at 2:05
  • 1
    For an actual answer see @tanius' reptyr reply. Mar 10, 2022 at 20:54
9

For running long lasting processes, I use screen, or byobu if you want a more friendly interface.

For screen, you can use:

screen [program] [args]

This will run [program] and its [args] inside a screen session. Once the program is finished, the session is automatically closed. If you wish to keep the session after your program runs, just run screen without any arguments and a new prompt will appear inside the session. CTRL+A+D detaches the terminal from the current session.

To re-attach to a previous session:

screen -r

If there is only one session open, it will reattach immediately. If multiple sessions are ongoing, it will ask you which one you want to attach to. If you know the session name, you can just add it as an argument to this command line.

Byobu is a nice improvement. It's based on screen, but provides a bar at the bottom that shows all current sessions as tabs and gives easier shortcuts to move around those. You can:

  • F2 start a new session
  • F3 move to the next session tab on the left
  • F4 move to the next session tab on the right
  • F8 give a friendly name to the current session tab
  • F9 opens a options menu
  • CTRL+A+D detaches all sessions from the terminal.

WORD OF ADVICE: avoid leaving a session opened with the user root. If anyone gains access to your terminal (locally or remotely), they can easily re-attach to an ongoing session and use your system as root. If needed, it's best to start a session using a common user and sudo indivudual command lines as necessary.

3
  • 2
    May I quote the OP: "Moral of the story is to use screen a lot". Apparently that was not the question here.
    – January
    Sep 28, 2012 at 12:13
  • Thanks for the writeup but January was correct.
    – user6658
    Sep 28, 2012 at 16:53
  • Use sudo screen <command> to setup a screen as root, which needs sudo access to reconnect to it. Far better than starting a screen normally, then changing to root within it. Aug 15, 2013 at 11:31
3

I was doing do-dist-upgrade via ssh from a laptop which went into suspend, hence Broken pipe. Upon going back into the machine I was seeing the upgrade-related processes still running, among which a whiptail asking me for input (which display manager to choose) and, relevantly, a root-owned SCREEN. I was able to do sudo su - and screen -r to attach to the session and, lo and behold, I have the whiptail dialog in front of me able to take input. I was able to resume the upgrade seamlessly.

Note: this was an upgrade from Ubuntu 14.04 to 16.04.

You must log in to answer this question.