Take the 2-minute tour ×
Ask Ubuntu is a question and answer site for Ubuntu users and developers. It's 100% free, no registration required.

I am building a MAAS environment with 4 physical machines.

  • 1 is running MAAS and Ubuntu 13.04

  • 3 are MAAS nodes running Ubuntu 12.10 (machine 0,1,2)

My problem is that after juju bootstrap only machine 0 ever get a running agent. All 3 nodes were setup on the same day.

MAAS web UI shows all nodes as "allocated to root".

You can ssh directly to machine 1 & 2 and there does not appear to be any trace of juju being installed on them. You can not juju ssh 1 or juju ssh 2

Here is juju status:

system@ubuntu1:~$ juju status
2013-11-22 23:19:40,553 INFO Connecting to environment...
2013-11-22 23:19:41,137 INFO Connected to environment.
machines:
  0:
    agent-state: running
    dns-name: 3ycae.ceph
    instance-id: /MAAS/api/1.0/nodes/node-d4ef6adc-522b-11e3-abf4-deadbeeefb1a/
    instance-state: unknown
  1:
    instance-id: pending
  2:
    instance-id: pending
  3:
    instance-id: pending
services: {}
2013-11-22 23:19:41,200 INFO 'status' command finished successfully

I have tried juju destroy-environment, the DNS is working correctly they can all ping each other by name. I have the firewall turned off on the MAAS host. I have tried deploying services but that doesn't help.

system@ubuntu1:~$ juju status
2013-11-22 23:23:54,514 INFO Connecting to environment...
2013-11-22 23:23:55,082 INFO Connected to environment.
machines:
  0:
    agent-state: running
    dns-name: 3ycae.ceph
    instance-id: /MAAS/api/1.0/nodes/node-d4ef6adc-522b-11e3-abf4-deadbeeefb1a/
    instance-state: unknown
  1:
    instance-id: pending
  2:
    instance-id: pending
  3:
    instance-id: pending
services:
  munin:
    charm: cs:precise/munin-3
    relations: {}
    units:
      munin/4:
        agent-state: pending
        machine: 1
        public-address: null
      munin/5:
        agent-state: pending
        machine: 2
        public-address: null
      munin/6:
        agent-state: pending
        machine: 3
        public-address: null
 2013-11-22 23:23:55,221 INFO 'status' command finished successfully

If I run juju -v ssh 1 I get this output, perhaps this will give you a clue.

system@ubuntu1:~$ juju -v ssh 1
2013-11-22 23:28:16,858 DEBUG Initializing juju ssh runtime
2013-11-22 23:28:16,862 INFO Connecting to environment...
2013-11-22 23:28:16,909 DEBUG Connecting to environment using 3ycae.ceph...
2013-11-22 23:28:16,909 DEBUG Spawning SSH process with remote_user="ubuntu"     remote_host="3ycae.ceph" remote_port="2181" local_port="39791".
2013-11-22 23:28:17,416:6182(0x7fa5364bd700):ZOO_INFO@log_env@658: Client environment:zookeeper.version=zookeeper C client 3.3.5
2013-11-22 23:28:17,416:6182(0x7fa5364bd700):ZOO_INFO@log_env@662: Client environment:host.name=ubuntu1
2013-11-22 23:28:17,416:6182(0x7fa5364bd700):ZOO_INFO@log_env@669: Client environment:os.name=Linux
2013-11-22 23:28:17,416:6182(0x7fa5364bd700):ZOO_INFO@log_env@670: Client environment:os.arch=3.8.0-29-generic
2013-11-22 23:28:17,416:6182(0x7fa5364bd700):ZOO_INFO@log_env@671: Client environment:os.version=#42~precise1-Ubuntu SMP Wed Aug 14 16:19:23 UTC 2013
2013-11-22 23:28:17,417:6182(0x7fa5364bd700):ZOO_INFO@log_env@679: Client environment:user.name=system
2013-11-22 23:28:17,417:6182(0x7fa5364bd700):ZOO_INFO@log_env@687: Client environment:user.home=/home/system
2013-11-22 23:28:17,417:6182(0x7fa5364bd700):ZOO_INFO@log_env@699: Client environment:user.dir=/home/system
2013-11-22 23:28:17,417:6182(0x7fa5364bd700):ZOO_INFO@zookeeper_init@727: Initiating   client connection, host=localhost:39791 sessionTimeout=10000 watcher=0x7fa5344886b0  sessionId=0 sessionPasswd=<null> context=0x25c4150 flags=0
2013-11-22 23:28:17,418:6182(0x7fa531263700):ZOO_INFO@check_events@1585: initiated connection to server [127.0.0.1:39791]
2013-11-22 23:28:17,427:6182(0x7fa531263700):ZOO_INFO@check_events@1632: session establishment complete on server [127.0.0.1:39791], sessionId=0x1427dc75f5d003a, negotiated timeout=10000
2013-11-22 23:28:17,431 DEBUG Environment is initialized.
2013-11-22 23:28:17,431 INFO Connected to environment.
2013-11-22 23:28:17,432 DEBUG Fetching machine address using juju machine id.
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1039, in _inlineCallbacks
result = g.send(result)
File "/usr/lib/python2.7/dist-packages/juju/control/utils.py", line 56, in get_ip_address_for_machine
provider_machine = yield provider.get_machine(instance_id)
File "/usr/lib/python2.7/dist-packages/juju/providers/common/base.py", line 178, in get_machine
d = self.get_machines([instance_id])
File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1181, in unwindGenerator
return _inlineCallbacks(None, gen, Deferred())
--- <exception caught here> ---
File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1039, in _inlineCallbacks
result = g.send(result)
File "/usr/lib/python2.7/dist-packages/juju/providers/maas/provider.py", line 91, in get_machines
instances = yield self.maas_client.get_nodes(instance_ids)
File "/usr/lib/python2.7/dist-packages/juju/providers/maas/maas.py", line 96, in  get_nodes
for resource_uri in resource_uris)
File "/usr/lib/python2.7/dist-packages/juju/providers/maas/maas.py", line 96, in <genexpr>
for resource_uri in resource_uris)
File "/usr/lib/python2.7/dist-packages/juju/providers/maas/maas.py", line 35, in   extract_system_id
match = _re_resource_uri.search(resource_uri)
exceptions.TypeError: expected string or buffer
2013-11-22 23:28:17,457 ERROR Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1039, in _inlineCallbacks
result = g.send(result)
File "/usr/lib/python2.7/dist-packages/juju/control/utils.py", line 56, in  get_ip_address_for_machine
provider_machine = yield provider.get_machine(instance_id)
File "/usr/lib/python2.7/dist-packages/juju/providers/common/base.py", line 178, in get_machine
d = self.get_machines([instance_id])
File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1181, in unwindGenerator
return _inlineCallbacks(None, gen, Deferred())
--- <exception caught here> ---
File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1039, in _inlineCallbacks
result = g.send(result)
File "/usr/lib/python2.7/dist-packages/juju/providers/maas/provider.py", line 91, in get_machines
instances = yield self.maas_client.get_nodes(instance_ids)
File "/usr/lib/python2.7/dist-packages/juju/providers/maas/maas.py", line 96, in get_nodes for resource_uri in resource_uris)
File "/usr/lib/python2.7/dist-packages/juju/providers/maas/maas.py", line 96, in <genexpr> for resource_uri in resource_uris)
File "/usr/lib/python2.7/dist-packages/juju/providers/maas/maas.py", line 35, in extract_system_id
match = _re_resource_uri.search(resource_uri)
exceptions.TypeError: expected string or buffer

expected string or buffer
2013-11-22 23:28:17,459 ERROR expected string or buffer
system@ubuntu1:~$ 

How can I resolve this ?

share|improve this question

2 Answers 2

Based on the traceback, you're using Juju 0.7 - the latest stable release of Juju is 1.16.3 - please destroy environment, upgrade juju to 1.16.3, then attempt to bootstrap again.

share|improve this answer

I now have the 3 node Ceph cluster working and can tear it down and rebuild it every time.

I am using the Ubunto 12.04 LTS & MAAS 1.2+bzr1373+dfsg-0ubuntu1~ & juju 0.5+bzr531-0ubuntu1.3.

This is what I learnt about doing this setup.

A) A 3 node Ceph cluster needs 5 machines.

  1. MAAS server
  2. Zookeeper server (This is always machine 0)

    3,4,5. The 3 nodes that will have Ceph deployed

B) The order of joining MAAS, Booting nodes, and performing juju bootstrap is super critical.

Here are my recommended steps...

Install MAAS and juju packages by following the standard install guide.

Get 1 machine ready to PXE boot. This is going to become machine 0 and will be the zookeeper.

PXE boot the machine and wait for it to boot and automatically shutdown. It will now show in the MAAS Gui.

From the MAAS Gui edit the node and Accept/Commission the node. It probably won't WOL so boot it manually or with "powerwake MAC_Address". Let it boot and wait for it to automatically shutdown. The MAAS Gui node status is now "ready".

Now you must run "juju bootstrap" on the MAAS server. Boot the node up again (DO NOT USE the MAAS Gui start button). The node will now install the OS and install zookeeper. When this is finished run juju status and machine 0 will be running. The MAAS Gui node status will be "allocated to root".

Now get your 3 CEPH nodes ready. PXE boot them and wait for them to boot and shutdown automatically. They will now be in MAAS. From the MAAS Gui accept/commision them. Boot them up again with "powerwake MAC_address". Wait for them to boot and shutdown automatically. They will now have MAAS status of ready and juju status will show the machines 1, 2, & 3 as instance-id: pending.

Now run "juju deploy --config ./ceph.yaml -n 3 ceph". At this point juju wants to provision 3 more machines.

Boot up each of the 3 CEPH nodes with "powerwake MAC_address" and they will each install the OS and receive the CEPH configuration.

This last step takes some time so resist the urge to destroy it and start again. Be patient.

Note# I found that the CEPH charm won't install the osd components if you have osd-journal: in the ceph.yaml file.

This is my ceph.yaml:

ceph:
  fsid: 6ca7da3e-4760-11e3-836d-ab53797f3654
  monitor-secret: AQDoDntSQErjFBAAMcVBPAiMgVNdqSO7tjjGDw==
  osd-devices: /dev/sdb /dev/sdc /dev/sdd /dev/sde

In the end you will have a juju status like this....

system@ubuntu1:~$ juju status
2013-11-27 10:13:08,997 INFO Connecting to environment...
2013-11-27 10:13:09,888 INFO Connected to environment.
machines:
  0:
    agent-state: running
    dns-name: rawpt.ceph
    instance-id: /MAAS/api/1.0/nodes/node-69ef3540-557e-11e3-91c1-deadbeeefb1a/
    instance-state: unknown
  1:
    agent-state: running
    dns-name: mm6c3.ceph
    instance-id: /MAAS/api/1.0/nodes/node-cf74149e-5583-11e3-8493-deadbeeefb1a/
    instance-state: unknown
  2:
    agent-state: running
    dns-name: tknpg.ceph
    instance-id: /MAAS/api/1.0/nodes/node-49682620-5587-11e3-8493-deadbeeefb1a/
    instance-state: unknown
  3:
    agent-state: running
    dns-name: 3ghwq.ceph
    instance-id: /MAAS/api/1.0/nodes/node-6da6cfd2-5587-11e3-91c1-deadbeeefb1a/
    instance-state: unknown
services:
  ceph:
    charm: cs:precise/ceph-18
    relations:
     mon:
      - ceph
    units:
      ceph/10:
        agent-state: started
        machine: 2
        public-address: tknpg.ceph
      ceph/11:
        agent-state: started
        machine: 3
        public-address: 3ghwq.ceph
      ceph/9:
        agent-state: started
        machine: 1
        public-address: mm6c3.ceph
2013-11-27 10:13:10,110 INFO 'status' command finished successfully


system@ubuntu1:~$ juju ssh 1 sudo ceph -s
2013-11-27 10:24:12,452 INFO Connecting to environment...
2013-11-27 10:24:13,026 INFO Connected to environment.
2013-11-27 10:24:13,059 INFO Connecting to machine 1 at mm6c3.ceph
   health HEALTH_OK
   monmap e2: 3 mons at {3ghwq=191.168.0.152:6789/0,mm6c3=191.168.0.150:6789/0,tknpg=191.168.0.151:6789/0}, election epoch 2, quorum 0,1,2 3ghwq,mm6c3,tknpg
   osdmap e25: 12 osds: 12 up, 12 in
    pgmap v120: 192 pgs: 192 active+clean; 0 bytes data, 12446 MB used, 44678 GB / 44690 GB avail
   mdsmap e1: 0/0/1 up
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.