4

In a host running Ubuntu 14.04.5 LTS I have an user named ci that can create an start unprivilaged lxc containers also running Ubuntu 14.04.5 LTS. The user has the subid range 200000-231071. The configuration file of such a container is:

# Distribution configuration
lxc.include = /usr/share/lxc/config/ubuntu.common.conf
lxc.include = /usr/share/lxc/config/ubuntu.userns.conf
lxc.arch = x86_64

# Nested
lxc.mount.auto = cgroup
lxc.aa_profile = lxc-container-default-with-nesting

# Container specific configuration
lxc.id_map = u 0 200000 65536
lxc.id_map = u 100000 265536 65536
lxc.id_map = g 0 200000 65536
lxc.id_map = g 100000 265536 65536
lxc.rootfs = /home/ci/.local/share/lxc/ci/rootfs
lxc.utsname = ci

# Network configuration
lxc.network.type = veth
lxc.network.flags = up
lxc.network.link = lxcbr0
lxc.network.hwaddr = 00:16:3e:dd:f1:99

The user can create and start the unprivilaged container without issues:

ci@host:~$ lxc-create -t download -n ci -- -d ubuntu -r trusty -a amd64
ci@host:~$ lxc-start -n ci -d
ci@host:~$ lxc-ls --fancy
    NAME  STATE    IPV4                 IPV6  AUTOSTART
    ---------------------------------------------------
    ci    RUNNING  10.0.3.75, 10.0.4.1  -     NO

In the host, cgmanager is running:

root@host ~ # ps ax | grep cgmanager
    382 ?        Ss     0:01 /sbin/cgmanager --sigstop -m name=systemd

In the unprivilaged container ci, cgproxy is running:

root@ci:~# ps ax | grep cgproxy
    288 ?        Ss     0:00 /sbin/cgproxy --sigstop

In the unprivilaged container ci, a user named jenkins with subid ranges 100000-65535 can create and start unprivilaged containers inside it, i.e. unprivilaged nested containers, but not without some tricks, which are:

  1. After logging in with ssh as user jenkins in the unprivilaged container ci, the result of cat /proc/self/cgroup is:

    jenkins@ci:~$ cat /proc/self/cgroup
        12:hugetlb:/user/1012.user/11.session/lxc/ci
        11:net_prio:/user/1012.user/11.session/lxc/ci
        10:perf_event:/user/1012.user/11.session/lxc/ci
        9:net_cls:/user/1012.user/11.session/lxc/ci
        8:freezer:/user/1012.user/11.session/lxc/ci
        7:devices:/user/1012.user/11.session/lxc/ci
        6:memory:/user/1012.user/11.session/lxc/ci
        5:blkio:/user/1012.user/11.session/lxc/ci
        4:name=systemd:/user/1012.user/11.session/lxc/ci/user/1012.user/11.session/lxc/ci/user/107.user/c1.session
        3:cpuacct:/user/1012.user/11.session/lxc/ci
        2:cpu:/user/1012.user/11.session/lxc/ci
        1:cpuset:/user/1012.user/11.session/lxc/ci
    
  2. At this point, jenkins can create a container, but cannot start it:

    jenkins@ci:~$ lxc-create -t download -n test -- -d ubuntu -r trusty -a amd64
    jenkins@ci:~$ lxc-start -n test
        lxc_container: cgmanager.c: lxc_cgmanager_create: 301 call to cgmanager_create_sync failed: invalid request
        lxc_container: cgmanager.c: lxc_cgmanager_create: 303 Failed to create hugetlb:lxc/test
        lxc_container: cgmanager.c: cgm_create: 650 Error creating cgroup hugetlb:lxc/test
        lxc_container: start.c: lxc_spawn: 891 failed creating cgroups
        lxc_container: start.c: __lxc_start: 1121 failed to spawn 'test'
        lxc_container: lxc_start.c: main: 341 The container failed to start.
        lxc_container: lxc_start.c: main: 345 Additional information can be obtained by setting the --logfile and --logpriority options.
    
  3. I issue in the container as root:

    restart systemd-logind
    
  4. Now as user jenkins in the container, I log out and log in again with ssh. The cgroup has changed and now I can create and run a container:

    jenkins@ci:~$ cat /proc/self/cgroup
        12:hugetlb:/user/1012.user/11.session/lxc/ci/user/1012.user/11.session/lxc/ci/user/107.user/c1.session
        11:net_prio:/user/1012.user/11.session/lxc/ci/user/1012.user/11.session/lxc/ci/user/107.user/c1.session
        10:perf_event:/user/1012.user/11.session/lxc/ci/user/1012.user/11.session/lxc/ci/user/107.user/c1.session
        9:net_cls:/user/1012.user/11.session/lxc/ci/user/1012.user/11.session/lxc/ci/user/107.user/c1.session
        8:freezer:/user/1012.user/11.session/lxc/ci/user/1012.user/11.session/lxc/ci/user/107.user/c1.session
        7:devices:/user/1012.user/11.session/lxc/ci/user/1012.user/11.session/lxc/ci/user/107.user/c1.session
        6:memory:/user/1012.user/11.session/lxc/ci/user/1012.user/11.session/lxc/ci/user/107.user/c1.session
        5:blkio:/user/1012.user/11.session/lxc/ci/user/1012.user/11.session/lxc/ci/user/107.user/c1.session
        4:name=systemd:/user/1012.user/11.session/lxc/ci/user/1012.user/11.session/lxc/ci/user/107.user/c1.session
        3:cpuacct:/user/1012.user/11.session/lxc/ci/user/1012.user/11.session/lxc/ci/user/107.user/c1.session
        2:cpu:/user/1012.user/11.session/lxc/ci/user/1012.user/11.session/lxc/ci/user/107.user/c1.session
        1:cpuset:/user/1012.user/11.session/lxc/ci/user/1012.user/11.session/lxc/ci/user/107.user/c1.session
    jenkins@ci:~$ lxc-create -t download -n test -- -d ubuntu -r trusty -a amd64
    jenkins@ci:~$ lxc-start -n test -d
    jenkins@ci:~$ lxc-ls --fancy
        NAME     STATE    IPV4       IPV6  AUTOSTART
        --------------------------------------------
        test     RUNNING  10.0.4.64  -     NO
    

First question: why do I need restart systemd-logind and how can I avoid typing it as root before being able to create nested unprivilaged containers?

In the container ci, I have created an init configuration file (an upstart conf file at /etc/init/jenkins.conf) to run the software Jenkins as user jenkins:

description "jenkins"

start on filesystem and static-network-up
stop on runlevel [016]

env USER="jenkins"
env GROUP="jenkins"
env HOME="/var/lib/jenkins"
env JENKINS_LOG="/var/log/jenkins"
env JENKINS_ROOT="/usr/share/jenkins"
env JENKINS_RUN="/var/run/jenkins"
env JENKINS_PIDFILE="jenkins.pid"

pre-start script
    test -f $JENKINS_ROOT/jenkins.war || { stop ; exit 0; }
    mkdir $JENKINS_RUN > /dev/null 2>&1  || true
    chown -R $USER:$GROUP $JENKINS_RUN || true
    mkdir $JENKINS_LOG > /dev/null 2>&1  || true
    chown -R $USER:$GROUP $JENKINS_LOG || true
end script

script
    . /etc/default/jenkins
    # export XDG_SESSION_ID="/run/user/`id -u $USER`"
    export HOME
    export USER
    export GROUP
    exec daemon --name=jenkins --foreground --inherit --user=$USER:$GROUP --pidfile=$JENKINS_RUN/$JENKINS_PIDFILE --output=$JENKINS_LOG -- $JAVA $JAVA_ARGS -jar $JENKINS_WAR $JENKINS_ARGS
end script

post-start script
    while [ ! -f $JENKINS_RUN/$JENKINS_PIDFILE ]; do sleep 1; done
    PID=$(cat $JENKINS_RUN/$JENKINS_PIDFILE)
    cgm create all $USER
    cgm chown all $USER $(id -u $USER) $(id -g $USER)
    # this need to be run in the jenkins job script:
    # cgm movepid all $USER $$
end script

# vim: ft=upstart

In the script that the process Jenkins issues to start a so called Jenkins' build, if I add the line:

cgm movepid all $USER $$

the script can create and start unprivilaged nested containers, being its cgroup:

+ cat /proc/self/cgroup
12:hugetlb:/user/1012.user/11.session/lxc/ci/jenkins
11:net_prio:/user/1012.user/11.session/lxc/ci/jenkins
10:perf_event:/user/1012.user/11.session/lxc/ci/jenkins
9:net_cls:/user/1012.user/11.session/lxc/ci/jenkins
8:freezer:/user/1012.user/11.session/lxc/ci/jenkins
7:devices:/user/1012.user/11.session/lxc/ci/jenkins
6:memory:/user/1012.user/11.session/lxc/ci/jenkins
5:blkio:/user/1012.user/11.session/lxc/ci/jenkins
4:name=systemd:/user/1012.user/11.session/lxc/ci/jenkins
3:cpuacct:/user/1012.user/11.session/lxc/ci/jenkins
2:cpu:/user/1012.user/11.session/lxc/ci/jenkins
1:cpuset:/user/1012.user/11.session/lxc/ci/jenkins

but a user jenkins logged in with ssh cannot stop the container created by the script. The following never finishes:

jenkins@ci:~$ lxc-stop -n test

Second question: how can I achieve that the user jenkins can stop any container created by the user jenkins from an init script like the above?

  • Your question is very advanced. It might get better exposure to the subject matter experts if you ask (or point to it) on the server team e-mail list. – Doug Smythies Aug 19 '16 at 15:17
  • What kind of init job is this? Is it an upstart user / session job? – BobTuckerman Nov 5 '16 at 22:37
  • @BobTuckerman the init job is a system one at /etc/init/jenkins.conf. I have clarified it editing the question. – Ivan Ogai Dec 5 '16 at 7:38
  • @DougSmythies I have just asked in the mailing list. – Ivan Ogai Dec 5 '16 at 10:21
  • @IvanOgai, I'm not such expert but detailed logs may bring additional helpful info. Could you share parent/nested container logs, before & after restart systemd-logind when trying to start nested one. Also logs for both of them when running lxc-stop -n test. You may post them to paste.ubuntu.com – user.dz Dec 16 '16 at 10:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.