MAAS node unable to resolve it's own hostname

Question

I've deployed a MAAS region/rack server, with the main eth interface connects to WAN and another connectes to a switch by using iptables as my MAAS-vlan with DHCP configured.

I've found myself unable to get storage information from both of my 2 machines (with different hardware), after hours of digging I've found that the name resolving has some error and the nodes were unable to resolve their own hostname when commissioning, which also made the commissioning process painfully long, since it's waiting name-resolving to time out most of the time. (that's a guess, but after I successfully logged into the box, ping golden-moose would take some 10 secs then throw an "unknown host" error)

the 00-maas-07-block-devices.err commissioning output reads:

sudo: unable to resolve host golden-moose: Connection timed out
sudo: unable to resolve host golden-moose: Connection timed out
sudo: unable to resolve host golden-moose: Connection timed out
sudo: unable to resolve host golden-moose: Connection timed out

I'm using MAAS Version 2.1.1+bzr5544-0ubuntu1 (16.04.1) and not sure how to debug this issue, please help, thanks.

The DNS service seems to be running OK, the nodes were able to resolve both external hosts and the .maas domain.

UPDATE

I've updated MAAS to 2.1.3, and same problem. After logging into an commissioning node (by the "Allow SSH access and prevent machine from powering off" option), I've found that the node was able to ping hostnames ONLY WITH ".maas" APPENDED. Which means the domainname wasn't properly set.

$ hostname -f
hostname: Name or service not known

$ domainname
(none)

The iptables rules seems working fine. The following commands all prints reasonable outputs (with non-zero packet counts)

$ sudo iptables -t raw -L -n -v
Chain PREROUTING (policy ACCEPT 645K packets, 185M bytes)
Chain OUTPUT (policy ACCEPT 411K packets, 1140M bytes)

$ sudo iptables -t nat -L -n -v
Chain PREROUTING (policy ACCEPT 73538 packets, 11M bytes)
Chain INPUT (policy ACCEPT 62414 packets, 9009K bytes)
Chain OUTPUT (policy ACCEPT 6585 packets, 493K bytes)
Chain POSTROUTING (policy ACCEPT 360 packets, 54084 bytes)

$ sudo iptables -t filter -L -n -v
Chain INPUT (policy ACCEPT 1772K packets, 875M bytes)
Chain FORWARD (policy DROP 694 packets, 185K bytes)
Chain OUTPUT (policy ACCEPT 1033K packets, 2318M bytes)

UPDATE - DNS dump

Using the tcpdump tool I've traced the node's DNS queries.

Typical node hostname queries by sudo look like the following (twice):

11:48:02.836710 IP (tos 0x0, ttl 64, id 53634, offset 0, flags [DF], proto UDP (17), length 57)
    <node-ip>.35343 > <maas-ip>.53: [udp sum ok] 8298+ A? pure-mammal. (29)
11:48:02.836750 IP (tos 0x0, ttl 64, id 53635, offset 0, flags [DF], proto UDP (17), length 57)
    <node-ip>.35343 > <maas-ip>.53: [udp sum ok] 36815+ AAAA? pure-mammal. (29)
11:48:02.836938 IP (tos 0x0, ttl 64, id 40343, offset 0, flags [none], proto UDP (17), length 132)
    <maas-ip>.53 > <node-ip>.35343: [bad udp cksum 0x71e4 -> 0x8095!] 36815 NXDomain q: AAAA? pure-mammal. 0/1/0 ns: . [2h34m56s] SOA a.root-servers.net. nstld.verisign-grs.com. 2017012101 1800 900 604800 86400 (104)
11:48:02.836945 IP (tos 0x0, ttl 64, id 40461, offset 0, flags [none], proto UDP (17), length 132)
    <maas-ip>.53 > <node-ip>.35343: [bad udp cksum 0x71e4 -> 0x0afb!] 8298 NXDomain q: A? pure-mammal. 0/1/0 ns: . [2h34m56s] SOA a.root-servers.net. nstld.verisign-grs.com. 2017012101 1800 900 604800 86400 (104)

Although I notice [bad udp cksum] bit, I have checked later that it wasn't affecting the result from the node.

A dig call with pure-mammal.maas from the commissioning node would result in log:

11:50:57.723037 IP (tos 0x0, ttl 64, id 24007, offset 0, flags [none], proto UDP (17), length 73)
    <node-ip>.53704 > <maas-ip>.53: [udp sum ok] 5376+ [1au] A? pure-mammal.maas. ar: . OPT UDPsize=4096 (45)
11:50:57.723321 IP (tos 0x0, ttl 64, id 5403, offset 0, flags [none], proto UDP (17), length 119)
    <maas-ip>.53 > <node-ip>.53704: [bad udp cksum 0x71d7 -> 0x8af0!] 5376* q: A? pure-mammal.maas. 1/1/2 pure-mammal.maas. [30s] A <node-ip> ns: maas. [30s] NS maas. ar: maas. [30s] A <maas-ip>, . OPT UDPsize=4096 (91)

This call results valid dig output from the node.

Final Update & Conclusion

While the hostname issue was indeed there, the problem lead to no storage configuration was something completely different.

After hours of checking and lots of advices from @mpontillo, I've finally made commissioning work. The surprise was the 2 of the 3 commissioning options, i.e. "Retain network configuration" and "Retain storage configuration". I checked those 2 every time, as I thought those are to "Retain" the information from the nodes. The storage config was read correctly after those unchecked.

Answer 1

First, I recommend that you update to MAAS 2.1.3, which is available in xenial-updates, and try the commissioning again. This will rule out any known issues.

Thinking about this problem, the Connection timed out message is what worries me the most. That means you're not getting a response from the DNS server, so I think this problem is very likely to be a DNS connectivity issue. To solve this, we might need to see the output of the following commands on your dual-homed MAAS server:

sudo iptables -t raw -L -n -v
sudo iptables -t nat -L -n -v
sudo iptables -t filter -L -n -v

If the firewall rules look good, I would then troubleshoot by commissioning the node with the Allow SSH access and prevent machine from powering off option. Then SSH in and use dig $(hostname -f) to verify that you can resolve the host from the commissioning node itself. You could try host $(hostname), too, which would test that the search path is working okay.

Then I would check /etc/bind/maas/named.conf.maas on the MAAS server to ensure that the network you are trying to reach MAAS from is in the list of trusted networks. (MAAS should automatically update this ACL.)

Finally, check the syslog on the MAAS server to make sure everything looks okay, such as grep named /var/log/syslog.

Somewhat related is bug #1087183, which talks about the fact that a standard Ubuntu install adds a line with the hostname to /etc/hosts, but in MAAS that has caused problems, so MAAS must rely on DNS.

Answer 2

During commissioning, resolv.conf has only a nameserver. When we deploy, it has a full search list, with the machine's name first, of course.

During commissioning, the machine is told its DNSDOMAIN, but it appears that the domain does not get into /etc/resolv.conf

I have filed Bug 1658750 for this issue.

For clarity, sudo failing to resolve the name results only in that warning message being printed: it does nothing else, and sudo does what you told it to. (It's trying to get the hostname so that it can compare it to any host-locked rules in sudoers, of which there are none.)