18

I have a number of LVM partitions, each containing an Ubuntu installation. Occasionally, I want to do an apt-get dist-upgrade, to update an installation to the most recent packages. I do this with chroot - the process is usually something like:

$ sudo mount /dev/local/chroot-0 /mnt/chroot-0
$ sudo chroot /mnt/chroot-0 sh -c 'apt-get update && apt-get dist-upgrade'
$ sudo umount /mnt/chroot-0

[ not shown: I also mount and unmount /mnt/chroot-0/{dev,sys,proc} as bind-mounts to the real /dev, /sys and /proc, as the dist-upgrade seems to expect these to be present ]

However, after upgrading to precise, this process no longer works - the final umount will fail because there are still open files on the /mnt/chroot-0 filesystem. lsof confirms that there are processes with open files in the chroot. These processes have been started during the dist-upgrade, I'm assuming this is because certain services in the chroot need to be restarted (eg, through service postgresql restart) after the package is upgraded.

So, I figure I need to tell upstart to stop all the services that are running within this chroot. Is there a way to reliably do this?

I've tried:

cat <<EOF | sudo chroot /mnt/chroot-0 /bin/sh
# stop 'initctl' services 
initctl list | awk '/start\/running/ {print \$1}' | xargs -n1 -r initctl stop
EOF

Where initctl list seems to do the right thing and only list processes that have been started in this particular root. I've tried adding this too, as suggested by Tuminoid:

cat <<EOF | sudo chroot /mnt/chroot-0 /bin/sh
# stop 'service' services
service --status-all 2>/dev/null |
    awk '/^ \[ \+ \]/ { print \$4}' |
    while read s; do service \$s stop; done
EOF

However, these doesn't seem to catch everything; processes that have daemonised and been reparented to PID 1 don't get stopped. I've also tried:

sudo chroot /mnt/chroot-0 telinit 0

But in this case, init doesn't distinguish between the separate roots and shuts down the entire machine.

So, is there any way to tell init to stop all processes in a particular chroot, so that I can safely unmount the filesystem? Does upstart have any facility to SIGTERM/SIGKILL all child processes (as would be done during regular shutdown) within a chroot?

1
  • This isn’t an answer to your actual question, but might be helpful: I recommend looking at the lxc package. lxc provides easy tools to start up and cleanly shut down instances in containers.
    – ion
    Jul 23, 2012 at 7:39

6 Answers 6

18
+200

I don't trust anything but the kernel to keep a sane state here, so I don't (ab)use init to get this job done, nor do I count on myself actually knowing what is or isn't mounted (some packages can mount extra filesystems, like binfmt_misc). So, for process slaughter, I use:

PREFIX=/mnt/chroot-0
FOUND=0

for ROOT in /proc/*/root; do
    LINK=$(readlink $ROOT)
    if [ "x$LINK" != "x" ]; then
        if [ "x${LINK:0:${#PREFIX}}" = "x$PREFIX" ]; then
            # this process is in the chroot...
            PID=$(basename $(dirname "$ROOT"))
            kill -9 "$PID"
            FOUND=1
        fi
    fi
done

if [ "x$FOUND" = "x1" ]; then
    # repeat the above, the script I'm cargo-culting this from just re-execs itself
fi

And for umounting chroots, I use:

PREFIX=/mnt/chroot-0
COUNT=0

while grep -q "$PREFIX" /proc/mounts; do
    COUNT=$(($COUNT+1))
    if [ $COUNT -ge 20 ]; then
        echo "failed to umount $PREFIX"
        if [ -x /usr/bin/lsof ]; then
            /usr/bin/lsof "$PREFIX"
        fi
        exit 1
    fi
    grep "$PREFIX" /proc/mounts | \
        cut -d\  -f2 | LANG=C sort -r | xargs -r -n 1 umount || sleep 1
done

As an addendum, I'd point out that approaching this as an init problem is probably the wrong way to look at it, unless you actually have an init in the chroot and a separate process space (ie: in the case of LXC containers). With a single init (outside the chroot), and a shared process space, this is no longer "init's problem", but rather just up to you to find the processes that happen to have the offending path, hence the above proc walk.

It's not clear from your initial post if these are fully-bootable systems that you're just upgrading externally (which is how I read it), or if they're chroots that you use for things like package builds. If it's the latter, you might also want a policy-rc.d in place (like the one dropped in by mk-sbuild) that just forbids init jobs starting in the first place. Obviously, that's not a sane solution if these are meant to be bootable systems as well.

5
  • They're bootable systems, but policy-rc.d looks like an interesting approach (I could simply remove it after interacting with the chroot). Does this affect both /etc/rc*.d- and /etc/init/*.conf-style jobs? Jul 23, 2012 at 8:03
  • Hm, evidently not: bugs.launchpad.net/ubuntu/+source/upstart/+bug/939105 Jul 23, 2012 at 8:23
  • Neither upstart nor sysvinit "consult policy-rc.d", it's invoke-rc.d that does so, which all postinst scripts are meant to use to interact with init jobs. In practice, it seems to DTRT, except in the case of broken packages (which should be fixed). Still, the above "purge with fire" scripting does the job, whether the issue is something slipping past policy, no policy being in place, or a long-running process of some other sort being left around (the major use-case for the buildds here are things backgrounded during the build itself or otherwise unparented from sbuild).
    – infinity
    Jul 23, 2012 at 9:01
  • 1
    One problem with trying to work around utpstart's chroot support. I'm fairly certain kill -9 will not prevent upstart from respawning the upstart job if it has respawn specified. So you really do still need to interrogate upstart from inside the chroot to find out if things are still running. I think this is pretty unfortunate, and we should have some way from outside chroots to kill off these jobs. That said I do see where the initctl list/awk/grep approach + yours should be complete.
    – SpamapS
    Jul 29, 2012 at 6:57
  • 1
    @SpamapS: good point - killing init jobs manually does indeed result in them being restarted. Would be great to be able to tell upstart to perform a chroot-specific shutdown, stopping defined jobs, and then killing any remaining reparented process that have a root directory within the chroot. Jul 30, 2012 at 8:34
0

You already identified the problem yourself: Some things run service ... during dist-upgrade and service isn't part of Upstart, but part of sysvinit. Add similar awk magic around service --status-all to stop sysvinit services as you used for Upstart services.

1
  • 3
    Ah, thanks. It's almost better, but that doesn't cover all services either. I've run sudo chroot /mnt/chroot-0 service --list-all and sudo chroot /mnt/chroot-0 initctl list, which both report no services running. However, /usr/bin/epmd (from erlang-base) is still running. Jul 11, 2012 at 13:42
0

I know this question is pretty old, but I think it is as relevant today as it was in 2012, and hopefully someone finds this code useful. I wrote the code for something I was doing, but thought I'd share it.

My code is different, but the ideas are very similar to @infinity (in fact - the only reason I now know about /proc/*/root is because of his answer - thanks @infinity!). I also added some cool additional functionality

#Kills any PID passed to it
#At first it tries nicely with SIGTERM
#After a timeout, it uses SIGKILL
KILL_PID()
{
        PROC_TO_KILL=$1

        #Make sure we have an arg to work with
        if [[ "$PROC_TO_KILL" == "" ]]
        then
                echo "KILL_PID: \$1 cannot be empty"
                return 1
        fi

        #Try to kill it nicely
        kill -0 $PROC_TO_KILL &>/dev/null && kill -15 $PROC_TO_KILL

        #Check every second for 5 seconds to see if $PROC_TO_KILL is dead
        WAIT_TIME=5

        #Do a quick check to see if it's still running
        #It usually takes a second, so this often doesn't help
        kill -0 $PROC_TO_KILL &>/dev/null &&
        for SEC in $(seq 1 $WAIT_TIME)
        do
                sleep 1

                if [[ "$SEC" != $WAIT_TIME ]]
                then
                        #If it's dead, exit
                        kill -0 $PROC_TO_KILL &>/dev/null || break
                else
                        #If time's up, kill it
                        kill -0 $PROC_TO_KILL &>/dev/null && kill -9 $PROC_TO_KILL
                fi
        done
}

Now you would do 2 things to make sure chroot can be unmounted:

Kill all processes that may be running in the chroot:

CHROOT=/mnt/chroot/

#Find processes who's root folder is actually the chroot
for ROOT in $(find /proc/*/root)
do
        #Check where the symlink is pointing to
        LINK=$(readlink -f $ROOT)

        #If it's pointing to the $CHROOT you set above, kill the process
        if echo $LINK | grep -q ${CHROOT%/}
        then
                PID=$(basename $(dirname "$ROOT"))
                KILL_PID $PID
        fi
done

Kill all processes that may be running outside of the chroot, but interfering with it (ex: if your chroot is /mnt/chroot and dd is writing to /mnt/chroot/testfile, /mnt/chroot will fail to unmount)

CHROOT=/mnt/chroot/

#Get a list of PIDs that are using $CHROOT for anything
PID_LIST=$(sudo lsof +D $CHROOT 2>/dev/null | tail -n+2 | tr -s ' ' | cut -d ' ' -f 2 | sort -nu)

#Kill all PIDs holding up unmounting $CHROOT
for PID in $PID_LIST
do
        KILL_PID $PID
done

Note: Run all code as root

Also, for a less complex version, replace KILL_PID with either kill -SIGTERM or kill -SIGKILL

0

jchroot: a chroot with more isolation.

After your command has been executed, any process started by the execution of this command will be killed, any IPC will be freed, any mount point will be unmounted. All clean!

schroot is not yet able to do this, but this is planned

I have tested it successfully in OpenVZ VPS, which can not use docker or lxc.

Please read the author's blog for the details:

https://vincent.bernat.im/en/blog/2011-jchroot-isolation.html

0

Just a contracted version of the chosen answer

Regex may be used to address multiple chroots at once.

Kill all processes in single or multiple $CHROOTS.

bash

for p in /proc/*/root; do [[ `readlink $p` =~ ^$CHROOTS ]] && kill ${p//[^0-9]/}; done

perl (faster)

perl -e "kill 15, map {/\d+/;\$&} grep {readlink =~ m:^$CHROOTS:} @ARGV" /proc/*/root

Umount single $CHROOT recursively.

bash

i=20; while ! umount -R "$CHROOT" && ((--i)); do sleep 1; done

Umount multiple $CHROOTS recursively.

bash

i=20; while ((i--))&&sleep 1&&a=`sed -nr "s:^\S+ ($CHROOTS) .*:\1:p" /etc/mtab`&&[[ $a ]]; do xargs umount -qR<<<"$a";done
-1

schroot: It has the feature of session management. When you stop the session its all processes are killed.

https://github.com/dnschneid/crouton/blob/master/host-bin/unmount-chroot: This scripts kill all the chroot process and unmounts all the mounted devices.

1
  • It might be helpful to explain how using schroot over chroot can help solve the OP's problem. Jul 1, 2014 at 23:58

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .