Take the 2-minute tour ×
Ask Ubuntu is a question and answer site for Ubuntu users and developers. It's 100% free, no registration required.

After a power outage our OpenStack installation went down. First step is trying to get MAAS back up and running, but I'm having trouble doing just that. The RC and the CC are running and can communcation (have network).

In my understanding the MAAS Region Controller API runs as a Django application via WSGI on Apache. Apache runs, but fails on loading the MAAS WSGI script (/usr/share/maas/wsgi.py). Hence the service and the API is never started. RabbitMQ and other services seem to be running on the RC. I have also tried to do:

dpkg-reconfigure maas-region-controller 

without any success.

The log below can be found in /var/log/apache/error.log.

Any ideas what I can try?

[Wed Jan 08 11:01:06.312804 2014] [mpm_event:notice] [pid 5725:tid 139810242742144] AH00491: caught SIGTERM, shutting down
[Wed Jan 08 11:01:06.326364 2014] [:error] [pid 5728:tid 139810242742144] mod_wsgi (pid=5728): Target WSGI script '/usr/share/maas/wsgi.py' cannot be loaded as Python module.
[Wed Jan 08 11:01:06.326607 2014] [:error] [pid 5728:tid 139810242742144] mod_wsgi (pid=5728): Exception occurred processing WSGI script '/usr/share/maas/wsgi.py'.
[Wed Jan 08 11:01:06.326846 2014] [:error] [pid 5728:tid 139810242742144] Traceback (most recent call last):
[Wed Jan 08 11:01:06.327054 2014] [:error] [pid 5728:tid 139810242742144]   File "/usr/share/maas/wsgi.py", line 32, in <module>
[Wed Jan 08 11:01:06.327424 2014] [:error] [pid 5728:tid 139810242742144]     start_up()
[Wed Jan 08 11:01:06.327611 2014] [:error] [pid 5728:tid 139810242742144]   File "/usr/lib/python2.7/dist-packages/maasserver/start_up.py", line 70, in start_up
[Wed Jan 08 11:01:06.327934 2014] [:error] [pid 5728:tid 139810242742144]     post_start_up()
[Wed Jan 08 11:01:06.328109 2014] [:error] [pid 5728:tid 139810242742144]   File "/usr/lib/python2.7/dist-packages/maasserver/start_up.py", line 110, in post_start_up
[Wed Jan 08 11:01:06.328320 2014] [:error] [pid 5728:tid 139810242742144]     NodeGroup.objects.refresh_workers()
[Wed Jan 08 11:01:06.328491 2014] [:error] [pid 5728:tid 139810242742144]   File "/usr/lib/python2.7/dist-packages/maasserver/models/nodegroup.py", line 116, in refresh_workers
[Wed Jan 08 11:01:06.328908 2014] [:error] [pid 5728:tid 139810242742144]     refresh_worker(nodegroup)
[Wed Jan 08 11:01:06.329456 2014] [:error] [pid 5727:tid 139810242742144] mod_wsgi (pid=5727): Target WSGI script '/usr/share/maas/wsgi.py' cannot be loaded as Python module.
[Wed Jan 08 11:01:06.329662 2014] [:error] [pid 5727:tid 139810242742144] mod_wsgi (pid=5727): Exception occurred processing WSGI script '/usr/share/maas/wsgi.py'.
[Wed Jan 08 11:01:06.329879 2014] [:error] [pid 5727:tid 139810242742144] Traceback (most recent call last):
[Wed Jan 08 11:01:06.330074 2014] [:error] [pid 5727:tid 139810242742144]   File "/usr/share/maas/wsgi.py", line 32, in <module>
[Wed Jan 08 11:01:06.330389 2014] [:error] [pid 5727:tid 139810242742144]     start_up()
[Wed Jan 08 11:01:06.330567 2014] [:error] [pid 5727:tid 139810242742144]   File "/usr/lib/python2.7/dist-packages/maasserver/start_up.py", line 70, in start_up
[Wed Jan 08 11:01:06.330879 2014] [:error] [pid 5727:tid 139810242742144]     post_start_up()
[Wed Jan 08 11:01:06.331050 2014] [:error] [pid 5727:tid 139810242742144]   File "/usr/lib/python2.7/dist-packages/maasserver/start_up.py", line 110, in post_start_up
[Wed Jan 08 11:01:06.331260 2014] [:error] [pid 5727:tid 139810242742144]     NodeGroup.objects.refresh_workers()
[Wed Jan 08 11:01:06.331426 2014] [:error] [pid 5727:tid 139810242742144]   File "/usr/lib/python2.7/dist-packages/maasserver/models/nodegroup.py", line 116, in refresh_workers
[Wed Jan 08 11:01:06.331831 2014] [:error] [pid 5727:tid 139810242742144]     refresh_worker(nodegroup)
[Wed Jan 08 11:01:06.332021 2014] [:error] [pid 5727:tid 139810242742144]   File "/usr/lib/python2.7/dist-packages/maasserver/refresh_worker.py", line 41, in refresh_worker
[Wed Jan 08 11:01:06.332207 2014] [:error] [pid 5728:tid 139810242742144]   File "/usr/lib/python2.7/dist-packages/maasserver/refresh_worker.py", line 41, in refresh_worker
[Wed Jan 08 11:01:06.332485 2014] [:error] [pid 5727:tid 139810242742144]     refresh_secrets.apply_async(queue=nodegroup.work_queue, kwargs=items)
[Wed Jan 08 11:01:06.332758 2014] [:error] [pid 5728:tid 139810242742144]     refresh_secrets.apply_async(queue=nodegroup.work_queue, kwargs=items)
[Wed Jan 08 11:01:06.332938 2014] [:error] [pid 5727:tid 139810242742144]   File "/usr/lib/python2.7/dist-packages/celery/app/task/__init__.py", line 459, in apply_async
[Wed Jan 08 11:01:06.333143 2014] [:error] [pid 5728:tid 139810242742144]   File "/usr/lib/python2.7/dist-packages/celery/app/task/__init__.py", line 459, in apply_async
[Wed Jan 08 11:01:06.333856 2014] [:error] [pid 5727:tid 139810242742144]     **options)
[Wed Jan 08 11:01:06.334453 2014] [:error] [pid 5728:tid 139810242742144]     **options)
[Wed Jan 08 11:01:06.334627 2014] [:error] [pid 5727:tid 139810242742144]   File "/usr/lib/python2.7/dist-packages/celery/app/amqp.py", line 196, in delay_task
[Wed Jan 08 11:01:06.334805 2014] [:error] [pid 5728:tid 139810242742144]   File "/usr/lib/python2.7/dist-packages/celery/app/amqp.py", line 196, in delay_task
[Wed Jan 08 11:01:06.335226 2014] [:error] [pid 5727:tid 139810242742144]     entity = self._declare_queue(queue, retry, _retry_policy)
[Wed Jan 08 11:01:06.335642 2014] [:error] [pid 5728:tid 139810242742144]     entity = self._declare_queue(queue, retry, _retry_policy)
[Wed Jan 08 11:01:06.335823 2014] [:error] [pid 5728:tid 139810242742144]   File "/usr/lib/python2.7/dist-packages/celery/app/amqp.py", line 172, in _declare_queue
[Wed Jan 08 11:01:06.336002 2014] [:error] [pid 5727:tid 139810242742144]   File "/usr/lib/python2.7/dist-packages/celery/app/amqp.py", line 172, in _declare_queue
[Wed Jan 08 11:01:06.336208 2014] [:error] [pid 5728:tid 139810242742144]     queue.declare()
[Wed Jan 08 11:01:06.336396 2014] [:error] [pid 5727:tid 139810242742144]     queue.declare()
[Wed Jan 08 11:01:06.336560 2014] [:error] [pid 5728:tid 139810242742144]   File "/usr/lib/python2.7/dist-packages/kombu/entity.py", line 492, in declare
[Wed Jan 08 11:01:06.336733 2014] [:error] [pid 5727:tid 139810242742144]   File "/usr/lib/python2.7/dist-packages/kombu/entity.py", line 492, in declare
[Wed Jan 08 11:01:06.337737 2014] [:error] [pid 5727:tid 139810242742144]     self.exchange.declare(nowait)
[Wed Jan 08 11:01:06.337941 2014] [:error] [pid 5728:tid 139810242742144]     self.exchange.declare(nowait)
[Wed Jan 08 11:01:06.338112 2014] [:error] [pid 5727:tid 139810242742144]   File "/usr/lib/python2.7/dist-packages/kombu/entity.py", line 162, in declare
[Wed Jan 08 11:01:06.338318 2014] [:error] [pid 5727:tid 139810242742144]     nowait=nowait, passive=passive,
[Wed Jan 08 11:01:06.338492 2014] [:error] [pid 5728:tid 139810242742144]   File "/usr/lib/python2.7/dist-packages/kombu/entity.py", line 162, in declare
[Wed Jan 08 11:01:06.338690 2014] [:error] [pid 5727:tid 139810242742144]   File "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 603, in exchange_declare
[Wed Jan 08 11:01:06.338897 2014] [:error] [pid 5728:tid 139810242742144]     nowait=nowait, passive=passive,
[Wed Jan 08 11:01:06.340001 2014] [:error] [pid 5727:tid 139810242742144]     (40, 11),  # Channel.exchange_declare_ok
[Wed Jan 08 11:01:06.340193 2014] [:error] [pid 5728:tid 139810242742144]   File "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 603, in exchange_declare
[Wed Jan 08 11:01:06.341389 2014] [:error] [pid 5727:tid 139810242742144]   File "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 69, in wait
[Wed Jan 08 11:01:06.341656 2014] [:error] [pid 5728:tid 139810242742144]     (40, 11),  # Channel.exchange_declare_ok
[Wed Jan 08 11:01:06.341972 2014] [:error] [pid 5727:tid 139810242742144]     self.channel_id, allowed_methods)
[Wed Jan 08 11:01:06.342292 2014] [:error] [pid 5728:tid 139810242742144]   File "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 69, in wait
[Wed Jan 08 11:01:06.342618 2014] [:error] [pid 5728:tid 139810242742144]     self.channel_id, allowed_methods)
[Wed Jan 08 11:01:06.342805 2014] [:error] [pid 5727:tid 139810242742144]   File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 204, in _wait_method
[Wed Jan 08 11:01:06.343462 2014] [:error] [pid 5727:tid 139810242742144]     self.method_reader.read_method()
[Wed Jan 08 11:01:06.343659 2014] [:error] [pid 5728:tid 139810242742144]   File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 204, in _wait_method
[Wed Jan 08 11:01:06.344309 2014] [:error] [pid 5728:tid 139810242742144]     self.method_reader.read_method()
[Wed Jan 08 11:01:06.344499 2014] [:error] [pid 5727:tid 139810242742144]   File "/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 187, in read_method
[Wed Jan 08 11:01:06.344883 2014] [:error] [pid 5727:tid 139810242742144]     raise m
[Wed Jan 08 11:01:06.345057 2014] [:error] [pid 5728:tid 139810242742144]   File "/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 187, in read_method
[Wed Jan 08 11:01:06.345561 2014] [:error] [pid 5728:tid 139810242742144]     raise m
[Wed Jan 08 11:01:06.345765 2014] [:error] [pid 5727:tid 139810242742144] error: [Errno 4] Interrupted system call
[Wed Jan 08 11:01:06.349404 2014] [:error] [pid 5728:tid 139810242742144] error: [Errno 4] Interrupted system call
[Wed Jan 08 11:01:07.375856 2014] [mpm_event:notice] [pid 7064:tid 140623783114624] AH00489: Apache/2.4.6 (Ubuntu) mod_wsgi/3.4 Python/2.7.5+ configured -- resuming normal operations
[Wed Jan 08 11:01:07.389375 2014] [core:notice] [pid 7064:tid 140623783114624] AH00094: Command line: '/usr/sbin/apache2'
share|improve this question

1 Answer 1

up vote 0 down vote accepted

Check /var/log/rabbitmq/rabbit@maas.log for a message like the following:

=INFO REPORT==== 20-Jan-2014::03:45:19 ===
Disk free limit set to 1000MB

=INFO REPORT==== 20-Jan-2014::03:45:19 ===
Disk free space insufficient. Free bytes:967696384 Limit:1000000000

=WARNING REPORT==== 20-Jan-2014::03:45:19 ===
disk resource limit alarm set on node rabbit@maas.

**********************************************************
*** Publishers will be blocked until this alarm clears ***
**********************************************************

If you see this message, free up disk space. If it's alerting on a memory resource limit, free up space in RAM or increase the RAM size of your machine.

share|improve this answer
    
I think that was it, thanks. –  jakob Jan 23 at 17:06

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.