I'm having issues launching juju on an lxc container through regular juju deploy commands:

juju bootstrap
juju deploy ubuntu

The ubuntu juju status continually says pending. The LXC starts, and I can even ssh into it if I get a hold of the IP address. But, the juju service fails to start in it. After some debugging, I have found that the upstart job fails to start.

What is going on?

Output from upstart service:

root@dpb-local-ubuntu-0:/home/ubuntu# ll /etc/init/juju*
-rw-r--r-- 1 root root 495 Feb  1 18:41 /etc/init/juju-ubuntu-0.conf
root@dpb-local-ubuntu-0:/home/ubuntu# service juju-ubuntu-0 start
start: Unknown job: juju-ubuntu-0

Manual starting of the script works (it even updates juju status correctly):

root@dpb-local-ubuntu-0:/home/ubuntu# export JUJU_ZOOKEEPER=""
root@dpb-local-ubuntu-0:/home/ubuntu# export JUJU_UNIT_NAME="ubuntu/0"
root@dpb-local-ubuntu-0:/home/ubuntu#  export JUJU_MACHINE_ID="None"
root@dpb-local-ubuntu-0:/home/ubuntu# export JUJU_HOME="/var/lib/juju"
root@dpb-local-ubuntu-0:/home/ubuntu# export JUJU_ENV_UUID="None"
root@dpb-local-ubuntu-0:/home/ubuntu# /usr/bin/python -m juju.agents.unit --nodaemon --logfile /var/log/juju/unit-ubuntu-0.log --session-file /var/run/juju/unit-ubuntu-0-agent.zksession

Here is the upstart config file:

description "Juju unit agent for ubuntu/0"
author "Juju Team <juju@lists.ubuntu.com>"

start on runlevel [2345]
stop on runlevel [!2345]

env JUJU_ENV_UUID="None"
env JUJU_HOME="/var/lib/juju"
env JUJU_UNIT_NAME="ubuntu/0"

exec /usr/bin/python -m juju.agents.unit --nodaemon --logfile /var/log/juju/unit-ubuntu-0.log --session-file /var/run/juju/unit-ubuntu-0-agent.zksession >> /var/log/juju/unit-ubuntu-0-output.log 2>&1

The following is from /var/log/cloud-init-output.log

Setting up juju ( ...
Processing triggers for libc-bin ...
ldconfig deferred processing now taking place
Processing triggers for python-support ...
start: Unknown job: juju-ubuntu-0
failed: /var/lib/cloud/instance/scripts/runcmd [1]
2013-02-01 18:41:37,603 - cc_scripts_user.py[WARNING]: failed to run-parts in /var/lib/cloud/instance/scripts
2013-02-01 18:41:37,604 - __init__.py[WARNING]: Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/cloudinit/CloudConfig/__init__.py", line 117, in run_cc_modules
    cc.handle(name, run_args, freq=freq)
  File "/usr/lib/python2.7/dist-packages/cloudinit/CloudConfig/__init__.py", line 78, in handle
    [name, self.cfg, self.cloud, cloudinit.log, args])
  File "/usr/lib/python2.7/dist-packages/cloudinit/__init__.py", line 326, in sem_and_run
  File "/usr/lib/python2.7/dist-packages/cloudinit/CloudConfig/cc_scripts_user.py", line 31, in handle
  File "/usr/lib/python2.7/dist-packages/cloudinit/util.py", line 227, in runparts
    raise RuntimeError('runparts: %i failures' % failed)
RuntimeError: runparts: 1 failures

2013-02-01 18:41:37,605 - __init__.py[ERROR]: config handling of scripts-user, None, [] failed

Full console log for reference: http://pastebin.ubuntu.com/1598256/

Requested output:

root@dpb-local-ubuntu-0:/var/log# date
Mon Feb  4 15:41:38 UTC 2013
root@dpb-local-ubuntu-0:/var/log# sudo start juju-ubuntu-0
start: Unknown job: juju-ubuntu-0
root@dpb-local-ubuntu-0:/var/log# ll /var/log/init
ls: cannot access /var/log/init: No such file or directory
root@dpb-local-ubuntu-0:/var/log# dmesg |grep init
[    0.000000] initial memory mapped: [mem 0x00000000-0x1fffffff]
[    0.000000] init_memory_mapping: [mem 0x00000000-0xbf7fffff]
[    0.000000] init_memory_mapping: [mem 0x100000000-0x33f7fffff]
[    0.000000] Memory: 12261276k/13623296k available (6718k kernel code, 1060604k absent, 301416k reserved, 6452k data, 932k init)
[    0.000124] Security Framework initialized
[    0.000167] AppArmor: AppArmor initialized
[    0.263902] devtmpfs: initialized
[    0.265782] Trying to unpack rootfs image as initramfs...
[    0.322530] SCSI subsystem initialized
[    0.335211] pnp: PnP ACPI init
[    0.599361] audit: initializing netlink socket (disabled)
[    0.599408] type=2000 audit(1359415390.492:1): initialized
[    0.616085] fuse init (API version 7.19)
[    0.723931] Freeing initrd memory: 21968k freed
[    0.816927] device-mapper: ioctl: 4.22.0-ioctl (2011-10-19) initialised: dm-devel@redhat.com
[    0.817336] ashmem: initialized
[    4.468141] device-mapper: dm-raid45: initialized v0.2594b
[    4.472486] async_tx: api initialized (async)
[   26.988510] init: eucalyptus-network (lo) main process (1170) killed by TERM signal
[   27.437140] init: failsafe main process (1166) killed by TERM signal
[   27.668253] Bluetooth: HCI device and connection manager initialized
[   27.668254] Bluetooth: HCI socket layer initialized
[   27.668254] Bluetooth: L2CAP socket layer initialized
[   27.668264] Bluetooth: SCO socket layer initialized
[   28.105061] Bluetooth: RFCOMM TTY layer initialized
[   28.105065] Bluetooth: RFCOMM socket layer initialized
[   56.880220] init: udev-fallback-graphics main process (2295) terminated with status 1
[   56.953337] init: gdm main process (2333) killed by TERM signal
[  128.387028] init: plymouth-stop pre-start process (4158) terminated with status 1

Further debugging:

root@dpb-local-ubuntu-2:~# ps -ef |grep juju
root     11444 11381  0 16:42 pts/0    00:00:00 grep --color=auto juju
root@dpb-local-ubuntu-2:~# initctl start juju-ubuntu-2
initctl: Unknown job: juju-ubuntu-2
root@dpb-local-ubuntu-2:~# initctl reload-configuration
root@dpb-local-ubuntu-2:~# initctl start juju-ubuntu-2
juju-ubuntu-2 start/running, process 11448
  • Can you please execute sudo start juju-ubuntu-0? Add any console output and content of /var/log/init/juju-ubuntu-0.log as well please. – Goran Miskovic Feb 3 '13 at 9:13
  • When Upstart fails to parse the job configuration file, you find it out exactly this way. There is no valid job for it to start, thus it will say start: Unknown job: juju-ubuntu-0. BUT, on my 12.10, that is very much valid Upstart job. Try looking at dmesg|grep "init:" for parse errors. – Tuminoid Feb 4 '13 at 12:17
  • Thanks @schkovich, I added that output. Is there another thing I could look at? – dpb Feb 4 '13 at 15:45
  • 1
    my bad @dbp. :( logs are in /var/log/upstart. however since you are still getting message start: Unknown job: juju-ubuntu-0 i guess that you will find nothing in logs. @Tuminoid is right when saying that there might be an error in job config file. try running init-checkconf /etc/init/juju-ubuntu-0.conf – Goran Miskovic Feb 4 '13 at 16:11
  • @schkovich -- hey, I got it running, though I'm at a loss as to why. Is there a way to fix this up without using initctl reload-configuration? This is on a juju-deployment and I'm fixing things manually. Look at "further debugging". Those steps are repeatable when the lxc node starts. – dpb Feb 4 '13 at 16:52

The problem ended up being the host was out of inotify resources, which made upstart unable to monitor the local filesystem for the new init script. On my system, I believe that crashplan (backup software) was the cuplript.

Increasing the amount of resources dedicated to inotify fixed the issue. Like this:

echo 1048576 > /proc/sys/fs/inotify/max_user_watches

# or in /etc/sysctl.conf to make it survive reboots:

Here are some good reference links:

Thanks all who helped answer this question!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.