So I ended up solving these problems myself, and if it will help others, I'm posting my steps here. I got this working perfectly, but it needed some tweaks for my setup which I'm going to share here in case it helps someone else. The nominal question I asked above was how I could get the LXC containers to come up on the MAAS LAN DHCP (10.207.39.0/24) network instead of the ROUTER WAN DHCP (192.168.1.0/24) network.
After much trial and error, it became clear that apparently I could not find the way to do this, and replies were not found, so I moved on to how to get my OpenStack working letting the LXC containers come up on their preferred WAN network.
In this scenario, the problem I ran into for quite some time was that the Juju Charms deployed to LXC containers were perpetually in "Pending" state in the Juju GUI and of course same status in "juju status". I hope I've captured all the steps that were required to completely fix this pending status problem. I'll update here if I missed anything, and I hope this helps others.
The basic strategy is to make these edits to several files in the /var/lib/lxc/juju-trusty-lxc-template/rootfs fileystem BEFORE you begin deploying LXC containers. These edits will work globally across all subsequently deployed containers. Just substitute your interface names and your LAN network in what follows, and it should work.
My maas server setup is:
Dell PowerEdge 2850, BIOS at latest A07 rev, and Dell BMC Firmware, v.1.83, A10 (these BIOS and Firmware upgrades are required)
15.10 ubuntu server edition on the maas server; all enlisted bare-metal nodes use the 14.04 trusty image from maas.
2 x 1Gb network ports
enp6s7 192.168.1.37 static IP
enp7s8 10.207.39.100 static IP
My maas enlistment bare-metals are also Dell PowerEdge 2850's. The Dell BMC IPMI is set to DHCP, and gets addresses on the 192.168.1.0/24 network, and the bare-metal maas enlisted nodes come up on the 10.207.39.0/24 network.
Tweak 1: Since I'm using two networks, I needed routing for the 10.207.39.0/24 network via the WAN on 192.168.1.0/24 in order to be able to download from the outside world to the maas-enlisted bare-metal LXC host(s) via "the internet". This was accomplished as follows by adding these rules to the maas server:
sudo iptables -A FORWARD -s 10.207.39.0/24 -o enp6s7 -j ACCEPT
sudo iptables -A FORWARD -d 10.207.39.0/24 -m state --state ESTABLISHED,RELATED -i enp6s7 -j ACCEPT
sudo iptables -t nat -A POSTROUTING -s 10.207.39.0/24 -o enp6s7 -j MASQUERADE
sudo iptables-save | sudo tee /etc/iptables.sav
Once the above rules are set, it should be possible to "nslookup google.com" and "ping -c 3 google.com" from the maas-enlisted bare-metal LXC host(s) that are on the 10.207.39.0/24 network.
Note 1: Here, enp6s7 is the "internet-connected" interface on the maas server on the 192.168.1.0/24 network (i.e. that goes to the router provided by my broadband internet service provider), and 10.207.39.0/24 is the private LAN network on the enp7s8 interface which is the network that maas and its' maas-enlisted bare-metal servers use.
Note 2: The rules are made permanent on the maas server across reboots by the following line added before the "exit 0" line in /etc/rc.local as shown below:
sudo iptables-restore < /etc/iptables.sav
exit 0
Note 3: Check that the rules were applied after reboot of the maas server as follows:
gstanden@ubuntu:~$ sudo iptables -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-A FORWARD -s 10.207.39.0/24 -o enp6s7 -j ACCEPT
-A FORWARD -d 10.207.39.0/24 -i enp6s7 -m state --state RELATED,ESTABLISHED -j ACCEPT
gstanden@ubuntu:~$
Tweak 2: Enable DNS lookups on the MAAS server. Use /etc/network/interfaces (because I tried doing this with /etc/dhcp/dhclient.conf and that would not work) but it works using /etc/network/interfaces. Add the "dns-nameservers" and the "dns-search" entries as shown below. Note, I found that they should be added to the enp6s7 specification; adding to the enp7s8 specification did not work for me.
gstanden@ubuntu:~$ cat /etc/network/interfaces
(comment) This file describes the network interfaces available on your system
(comment) and how to activate them. For more information, see interfaces(5).
source /etc/network/interfaces.d/*
(comment) The loopback network interface
auto lo
iface lo inet loopback
(comment) The primary network interface
auto enp6s7
iface enp6s7 inet static
address 192.168.1.37
network 192.168.1.0
netmask 255.255.255.0
gateway 192.168.1.1
broadcast 192.168.1.255
dns-search maas <-- add this entry
dns-nameservers 10.207.39.100 <-- add this entry
(comment) The MAAS network interface
auto enp7s8
iface enp7s8 inet static
address 10.207.39.100
network 10.207.39.0
netmask 255.255.255.0
gateway 10.207.39.100
broadcast 10.207.39.255
gstanden@ubuntu:~$
There is one other tweak needed on the maas server: Edit /etc/sysctl.conf on the maas controller server (not the lxc bare-metal host) such that the following command gives result as shown:
gstanden@ubuntu:~$ sudo sysctl -p
net.ipv4.ip_forward = 1
gstanden@ubuntu:~$
Note 1: This will allow nslookups and pings of enlisted bare-metal servers on the maas node in the 10.207.39.0/24 network that belong to this maas to work.
Tweak 3: Several tweaks are needed to establish robust networking services inside the LXC containers themselves. These tweaks to the lxc template will fix problems with downloading the juju tools from the bare-metal lxc host, and will fix problems resolving the WAN (192.168.1.0/24 addresses, and will fix problems resolving addresses on the 10.207.39.0/24 network).
Note 1: The problem that was encountered is/was that no matter how the /var/lib/lxc/machine/config file is tweaked, containers ALWAYS get an address on the 192.168.1.0/24 network, even though they are using the juju-br0 bridge which has a 10.207.39.0/24 address! (fun, huh?); but in this two-network LAN/WAN setup, what is needed is ultimately networking to both networks so in this scenario routing is used.
Tweak 3a: ssh ubuntu@portly-legs (portly-legs is the enlistment name that maas gave to my maas bare-metal lxc host)
sudo su -
cd /var/lib/lxc/juju-trusty-lxc-template/rootfs/etc/network/interfaces.d/
vi eth0.cfg and add the following lines to the eth0.cfg file (add the "up ip route..." and "dns-search maas" lines).
auto eth0
iface eth0 inet dhcp
up ip route add 10.207.39.0/24 via 192.168.1.37
dns-search maas
The ip route line will establish ssh / ping / scp connectivity etc. from the lxc containers on 192.168.1.0/24 (eth0 in the LXC container) to the maas-enlisted bare-metal lxc-host on the 10.207.39.0/24 network. The "dns-search maas" line is part of how maas DNS lookups are made available inside the LXC containers so that full DNS resolution including the maas network will work. This is especially needed when the lxc containers on the maas-enlisted bare-metal lxc host go to get juju tools - they lxc containers will try to pull this from the bare-metal lxc host (in this example it happens to be 10.207.39.152 which is portly-legs.maas) and without this routing, the tools pull from the lxc host at .152 will fail. You can what this succeed/fail by logging into the lxc container and doing "tail -f /var/log/cloud-init.log" and you will see the tools pull either failing or succeeding.
Tweak 3b: Edit the /var/lib/lxc/juju-trusty-lxc-template/rootfs/etc/dhcp/dhclient.conf file and add this line (you can uncomment the example line and modify it):
prepend domain-name-servers 10.207.39.100;
Note 1: This could probably alternatively be done in /var/lib/lxc/juju-trusty-lxc-template/rootfs/etc/network/interfaces.d/eth0.cfg using a "dns-nameservers 10.207.39.100" entry, but I've used dhclient.conf in this case.
I think this is all the tweaks. Once all these tweaks have been made, one can start deploying openstack components to LXC containers in this dual-network setup and they come out of pending state and into ready state as fast as flapjacks off the griddle! Please let me know if I missed any steps, and I will try to double-check these also. But I think that's it.
Once all of these steps are done, try deploying an openstack component to LXC and monitor the progress:
(from terminal on the maas server as the user who owns the maas deployment): ssh ubuntu@(ip address of a container)
on the container: sudo su -
on the container: tail -f /var/log/cloud-init.log
and/or
tail -f /var/log/cloud-init-output.log
If you see message like this, you are probably all good:
Attempt 1 to download tools from -ttps://10.207.39.152:17070/tools/1.25.3-trusty-amd64...
+ curl -sSfw tools from %{url_effective} downloaded: HTTP %{http_code}; time %{time_total}s; size %{size_download} bytes; speed %{speed_download} bytes/s --noproxy * --insecure -o /var/lib/juju/tools/1.25.3-trusty-amd64/tools.tar.gz -ttps://10.207.39.152:17070/tools/1.25.3-trusty-amd64
tools from -ttps://10.207.39.152:17070/tools/1.25.3-trusty-amd64 downloaded: HTTP 200; time 1.698s; size 18722171 bytes; speed 11026214.000 bytes/s + echo Tools downloaded successfully.
Tools downloaded successfully.
When there are issues with networking in the LXC containers, this will fail on multiple attempts. The above tweaks should completely fix those issues, and will also provide inside the LXC container full DNS lookups for bare-metal maas enlisted servers on the maas network (the LXC hosts and single product hosts such as nova-compute). Containers should go to fully deployed ready status in less than 5 minutes each. I deploy them one-at-at-time and I wait for each one to reach ready status. YMMV, HTH, Gilbert Standen, St. Louis, MO Feb. 11, 2016, 1:25 PM CT
