0

After many issues (refer to question https://askubuntu.com/questions/601092/issue-with-maas-cluster-controller-no-enlistment) I have a new problem.

Basically after commissioning installation fails. I really do not know what is going on. First what is weird is that installation downloads image form the region controller instead of the cluster controller. The result is that installation process takes awfully long. Actually this is why we installed a standalone cluster controller. After that I receive a couple of "Failed to query node's BMC" before Deployment is marked failed.

Anyone can help on this issue?

So far my findings are installation works neither on a second cluster interface (https://askubuntu.com/questions/569306/issue-with-maas-and-second-cluster-interface) neither on standalone cluster controller, which is pretty disappointing (and this maybe because region controller is used during installation when it would seem logical that cluster controller is used instead)...

My version of maas is 1.7.1+bzr3341-0ubuntu1~trusty1.

Help would be very much appreciated.

Edit

To give more details, the architecture is as per below. I also give the event list. I wonder too if it is important that domain name can be resolved as in the event I can see the given domain name of the node (in other words node are provisioned with domain name name.maas but are also edited to be registered as name.domain). With a basic Region Controller and Cluster Controller this works well and I do no see why it would not work with a second interface or a standalone Cluster Controller. I hope this is not too confusing.

Site 1: MAAS Region Controller + MAAS Cluster Controller with two interfaces/subnets. I am unable to provision nodes on the second interface (the one not defined for the API).

Site 2: MAAS Cluster Controller with one interface/subnet. Cluster is registered with Region Controller and images have been downloaded from Region Controller after name resolution correctly implemented (I still do not know why this is needed to have the cluster in sync with images). Without pserv.yaml updated with generator using the IP of the MAAS Region Controller (see my other question https://askubuntu.com/questions/601092/issue-with-maas-cluster-controller-no-enlistment), it is impossible to have nodes enlisted. With it enlistment works fine, commissioning too but subsequent deployment seem to download what is needed from Region Controller). Installation takes a long time and fails with the details below.

INFO    Thu, 26 March 2015 18:54:40 Installation complete — Node disabled netboot   
WARNING Thu, 26 March 2015 18:50:50 Failed to query node's BMC — Node could not be queried node-e7391f8a-d3d5-11e4-aaa5-5254005cf640 (m6.domain) virsh failed with return code 1: Failed to login to virsh console. 
WARNING Thu, 26 March 2015 18:45:50 Failed to query node's BMC — Node could not be queried node-e7391f8a-d3d5-11e4-aaa5-5254005cf640 (m6.domain) virsh failed with return code 1: Failed to login to virsh console. 
WARNING Thu, 26 March 2015 18:40:50 Failed to query node's BMC — Node could not be queried node-e7391f8a-d3d5-11e4-aaa5-5254005cf640 (m6.domain) virsh failed with return code 1: Failed to login to virsh console. 
WARNING Thu, 26 March 2015 18:38:47 Failed to query node's BMC — Node could not be queried node-e7391f8a-d3d5-11e4-aaa5-5254005cf640 (m6.domain) virsh failed with return code 1: Failed to login to virsh console. 
INFO    Thu, 26 March 2015 18:36:37 Node changed status — From 'Deploying' to 'Failed deployment'   
WARNING Thu, 26 March 2015 18:35:50 Failed to query node's BMC — Node could not be queried node-e7391f8a-d3d5-11e4-aaa5-5254005cf640 (m6.domain) virsh failed with return code 1: Failed to login to virsh console. 
WARNING Thu, 26 March 2015 18:30:50 Failed to query node's BMC — Node could not be queried node-e7391f8a-d3d5-11e4-aaa5-5254005cf640 (m6.domain) virsh failed with return code 1: Failed to login to virsh console. 
WARNING Thu, 26 March 2015 18:20:50 Failed to query node's BMC — Node could not be queried node-e7391f8a-d3d5-11e4-aaa5-5254005cf640 (m6.domain) virsh failed with return code 1: Failed to login to virsh console. 
WARNING Thu, 26 March 2015 18:15:50 Failed to query node's BMC — Node could not be queried node-e7391f8a-d3d5-11e4-aaa5-5254005cf640 (m6.domain) virsh failed with return code 1: Failed to login to virsh console. 
WARNING Thu, 26 March 2015 18:05:50 Failed to query node's BMC — Node could not be queried node-e7391f8a-d3d5-11e4-aaa5-5254005cf640 (m6.domain) virsh failed with return code 1: Failed to login to virsh console. 
WARNING Thu, 26 March 2015 18:00:50 Failed to query node's BMC — Node could not be queried node-e7391f8a-d3d5-11e4-aaa5-5254005cf640 (m6.domain) virsh failed with return code 1: Failed to login to virsh console.
DEBUG   Thu, 26 March 2015 17:56:47 TFTP Request — ubuntu/amd64/generic/trusty/release/boot-initrd
DEBUG   Thu, 26 March 2015 17:56:46 TFTP Request - ubuntu/amd64/generic/trusty/release/boot-kernel  
DEBUG   Thu, 26 March 2015 17:56:46 TFTP Request — ifcpu64.c32  
INFO    Thu, 26 March 2015 17:56:46 PXE Request — curtin install    
INFO    Thu, 26 March 2015 17:56:46 PXE Request — curtin install    
DEBUG   Thu, 26 March 2015 17:56:46 TFTP Request — pxelinux.cfg/01-52-54-00-15-a9-ed    
DEBUG   Thu, 26 March 2015 17:56:46 TFTP Request — pxelinux.cfg/01-52-54-00-15-a9-ed    
DEBUG   Thu, 26 March 2015 17:56:46 TFTP Request — pxelinux.cfg/456decab-b186-e9dd-16d5-49e72359e6c3    
DEBUG   Thu, 26 March 2015 17:56:46 TFTP Request — pxelinux.0   
INFO    Thu, 26 March 2015 17:56:43 Node powered on 
INFO    Thu, 26 March 2015 17:56:38 Powering node on    
INFO    Thu, 26 March 2015 17:56:37 Node changed status — From 'Allocated' to 'Deploying'   
INFO    Thu, 26 March 2015 17:56:37 Node changed status — From 'Ready' to 'Allocated' (to root) 
INFO    Thu, 26 March 2015 17:52:44 Node changed status — From 'Commissioning' to 'Ready'

Edit

I wonder if there is something common with this old bug https://bugs.launchpad.net/maas/+bug/1081701.

Indeed I can see the following when hitting the preseed button of the node that failed to install.

#cloud-config
datasource:
  MAAS: {consumer_key: <CONSUMER_KEY>, metadata_url: 'http://<MAAS_REGION_CONTROLLER_IP>/MAAS/metadata/',
token_key: <TOKEN_KEY>, token_secret: <TOKEN_SECRET>}

In the above do I need to have the Cluster Controller IP instead?

Note the same applies for the case of the second interface not working but here I would expect the other interface to be reachable as I think I got rid of all the RP filtering issues, which created problems at some point. However, maybe the IP in preseed should be the one of the Controller interface on the subnet.

Many thanks in advance for your help.

Edit

After giving up for some weeks I have upgraded to 1.8. Same thing happen and I cannot understand why. Enlistment works with remote cluster controller but after commissioning installation uses files from the region/cluster controller not from the remote cluster controller and node is flagged "Deployment failed".

Help would be appreciated.

9
  • In your original problem, you mentioned that the nodes were on a different subnet from the cluster controller. Is that still the case? – rmustakos Mar 26 '15 at 20:49
  • Hello, many thanks for your interest. To give you the full picture I am updating the question with the architecture. – sba Mar 27 '15 at 11:56
  • Just going over some issues I've had - if you had to change the cluster controller IP by hand, I bet you did not get it everywhere. Try this and enter your cluster controller IP sudo dpkg-reconfigure maas-region-controller Then this and set your region controller IP sudo dpkg-reconfigure maas-cluster-controller I suspect, if the deployment is going to the region, that the nodes think the region controller IP is the cluster controller IP. – rmustakos Mar 30 '15 at 18:27
  • Many thanks for your answer. I am not sure why you want me to enter the cluster controller IP address after dpkg-reconfigure maas-region-controller. According to documentation that should be the API address i.e. region controller isn't it. The same applies for dpkg-reconfigure maas-cluster-controller where the IP of the region controller needs to be inserted. However your comment lead me to wonder how the IP of the cluster controller inteface is fetched in the setup. To me that should be from the edition of Edit Cluster Contoller > Interfaces, which is here properly set too... – sba Mar 31 '15 at 14:27
  • I agree that it should be obvious, with one textbox to rule them all, but, having rebuilt the MaaS 10 or 15 times over the last 8 weeks, I can tell you it isn't, but I needed to document a step by step, break it down Barney style plan of how rebuild it in the event I get hit by a truck. For me, my region and cluster controllers are the same machine. I don't know all the places the IPs get placed, but the region and cluster IPs are stored in some places that I could not find, and which only get changed when you do a dpkg-reconfigure. – rmustakos Apr 2 '15 at 1:26
1

Hurray.

I finally resolved the issue with my remote cluster controller just after editing my post and this by accident.

The solution is as following.

When creating the remote cluster controller (named Cluster 2 in my case) add a DNS zone different that for the cluster controller associated with the region controller (named Cluster 1 in my case). By default it is maas so change it to maas2 (so maas for Cluster 1 and maas2 for Cluster 2).

After new node is enlisted when editing for Cluster pick "maas2" (in the list box instead of having maas and mass you have maas and maas2). This way to name cluster with zone name is odd to me but this is the way it works.

Now commission and then install. Boom, installation picks the right cluster controllers and only takes few seconds and most important is successful.

I hope this helps somebody as it really blocked me for several weeks.

I still need to find how to have the remote cluster controller acting as a name server as for now the region controller acts as a name serve although the zone.maas2 file seem to indicate the remote cluster controller as a name server too.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.