1

Setup is as follows, workstation <-> Maas server <-> 3 VM's - Instances

Maas Server and the VM's are deployed in vSphere.

The Maas server's eth1 has 10.4.x.x which is acting as DHCP manager for the VM's. My workstation's ip is 10.x.x.x and I am able to connect to the Maas server and the VM's via ssh. I have one dedicated compute node and the network node has two interfaces. eth0 (br0) has an ip-address from the maas-server and eth1 (br-ex) gets a publicly accessible ip address from 192.168.x.x range( external network )

The problem is I cannot connect to the instances directly from my workstation or any other terminal in our network. I have identified the problem to be within the router namespace. I cannot ping 8.8.8.8 from the neutron router namespace. It is able to ping br-ex, but shows 'destination unreachable' when I ping 192.168.x.1. Please have a look at the output of the following commands,

ip netns

qrouter-1ff7cd3f-7d73-4c7a-a9aa-0f8efb14a718

ip netns exec qrouter-1ff7cd3f-7d73-4c7a-a9aa-0f8efb14a718 ifconfig

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:65536 Metric:1 RX packets:16 errors:0 dropped:0 overruns:0 frame:0 TX packets:16 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:1776 (1.7 KB) TX bytes:1776 (1.7 KB)

qg-80014d1d-df Link encap:Ethernet HWaddr fa:16:3e:b8:62:2e
inet addr:192.168.x.x Bcast:192.168.x.255 Mask:255.255.255.0 inet6 addr: fe80::f816:3eff:feb8:622e/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:21036 errors:0 dropped:0 overruns:0 frame:0 TX packets:3095 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:4857288 (4.8 MB) TX bytes:1047570 (1.0 MB)

ip netns exec qrouter-1ff7cd3f-7d73-4c7a-a9aa-0f8efb14a718 ip route list

default via 192.168.x.1 dev qg-80014d1d-df

192.168.x.0/24 dev qg-80014d1d-df proto kernel scope link src 192.168.x.x

I have seen neutron setups where the router interface( qg-80014d1d-df )is included in the bridge br-ex. Also the diagram at openstack.org for neutron troubleshooting shows that br-ex should contain a 'qg' interface.

enter image description here

But in my node it shows different,

ovs-vsctl show

f50f0af5-a5ab-4ab8-935a-9eaf71b40eea

Bridge br-int

    fail_mode: secure
    Port br-int
        Interface br-int
            type: internal
    Port patch-tun
        Interface patch-tun
            type: patch
            options: {peer=patch-int}

Bridge br-ex
    Port "tap80014d1d-df"
        Interface "tap80014d1d-df"
    Port br-ex
        Interface br-ex
            type: internal
    Port "eth1"
        Interface "eth1"

Bridge br-tun
    Port br-tun
        Interface br-tun
            type: internal
    Port patch-int
        Interface patch-int
            type: patch
            options: {peer=patch-tun}
    Port "gre-0a04546a"
        Interface "gre-0a04546a"
            type: gre
            options: {in_key=flow, local_ip="10.4.x.x", out_key=flow, remote_ip="10.4.x.x"}
ovs_version: "2.0.2"

Here br-ex does get an additional interface but that is 'tap'. The numeric value for this 'tap' and 'qg' interface are same though. I have checked the log files on both the compute node and controller, but did not find anything significant. I can spawn instances and assign them floating ip's. I am also able to ping br-ex, but not the router interface within it. Not sure if the problem is caused by the 'tap' interface but there is no flow between the router namespace and outside world. Another thought came to my mind if the problem is caused by VMware vSphere !!!

One thing should be mentioned, while posting the outputs there is only external network and a router to check connectivity. There is no instances or internal network.

I am stuck on this thing like forever. Any suggestion would be highly appreciated.

0

I have very similar setup, so I will show you my configuration, maybe this will help you.

All nodes are deployed with maas and juju and each node has two interfaces:

  • eth0 - wake on lan, boot on lan, maas network - 10.5.0.0/16, gw 10.5.0.1
  • eth1 - external company network - 172.16.62.0/24 gw 172.16.62.254

I have installed Openstack with help of this tree guides

  1. http://insights.ubuntu.com/wp-content/uploads/UCD-latest.pdf?utm_source=Ubuntu%20Cloud%20documentation%20%E2%80%93%2014.04%20LTS&utm_medium=download+link&utm_content=

  2. http://linux.dell.com/files/whitepapers/Deploying_Workloads_With_Juju_And_MAAS-14.04LTS-Edition.pdf

  3. h20564.www2.hp.com/hpsc/doc/public/display?docId=c04330703 (I can't paste it as a link, because of my low askubuntu reputation)

My network setup on normal nodes:

#/etc/network/interfaces
auto lo

auto eth0    
iface eth0 inet manual

auto juju-br0
iface juju-br0 inet dhcp
    bridge_ports eth0


auto eth1   
iface eth1 inet static
    address 172.16.62.10
    netmask 255.255.255.0
    #gateway commented
    #gateway 172.16.62.254 


$route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.5.0.1        0.0.0.0         UG    0      0        0 juju-br0
10.5.0.0        0.0.0.0         255.255.0.0     U     0      0        0 juju-br0
172.16.62.0     0.0.0.0         255.255.255.0   U     0      0        0 eth1

on network neutron(quantum) node:

#/etc/network/interfaces
auto lo

auto eth0
iface eth0 inet manual

auto juju-br0
iface juju-br0 inet dhcp
    bridge_ports eth0

auto eth1
iface eth1 inet manual
        up ip link set dev $IFACE up
        down ip link set dev $IFACE down


$ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.5.0.1        0.0.0.0         UG    0      0        0 juju-br0
10.5.0.0        0.0.0.0         255.255.0.0     U     0      0        0 juju-br0

on network I have also edit /etc/sysctl.conf and uncomment these lines

net.ipv4.ip_forward=1
net.ipv4.conf.all.rp_filter=1
net.ipv4.conf.default.rp_filter=1

then relod changes

# sysctl -p

After enlisting the nodes in maas I start to deploy juju Openstack charms.

Post installation open stack configuration via Horizon:

As Admin I did:

  1. upload cirros image,
  2. create new tenant and user
  3. create new external network in my company range 172.16.62.192/26, but in a such a way that didn't overlap with normal dhcp address pull which is 172.16.62.10 - 172.16.62.100
  4. create router and set gatway to external network

As user I did:

  1. create new private network 192.168.0.1
  2. add new interface to router (created in previous steps)
  3. boot new instance
  4. change the default security group and allow for ICMP(ping) and ssh connections

Post deploy network configuration look like this:

#list of ovs ports
ubuntu@fair-stone:~$ sudo ovs-vsctl show
b4a92be0-56d3-4cd5-b113-9657c21daf8a
    Bridge br-ex
        Port br-ex
            Interface br-ex
                type: internal
        Port "tapf199827f-4f" #interface appears after creation new 
            Interface "tapf199827f-4f"
        Port "eth1"
            Interface "eth1"

    Bridge br-tun
        Port br-tun
            Interface br-tun
                type: internal
        Port "gre-0a050104"
            Interface "gre-0a050104"
                type: gre
                options: {in_key=flow, local_ip="10.5.1.1", out_key=flow, remote_ip="10.5.1.4"}
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
    Bridge br-int
        fail_mode: secure
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port "tap08438cff-fc"  
            tag: 1
            Interface "tap08438cff-fc" 
        Port "tap592b1f29-da"
            tag: 1
            Interface "tap592b1f29-da" 
        Port br-int
            Interface br-int
                type: internal
    ovs_version: "2.0.2"

Networking namespaces

ubuntu@fair-stone:~$ ip netns list
qdhcp-9d699ab8-940b-478e-ac98-ba3fd38e5d9d
qrouter-a2e0f664-e969-4df1-a2df-16adfbe82cf3

Interface list in namespace qrouter-xxx

ubuntu@fair-stone:~$ sudo ip netns exec qrouter-a2e0f664-e969-4df1-a2df-16adfbe82cf3 ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: qr-08438cff-fc: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether fa:16:3e:61:dd:b7 brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.1/24 brd 192.168.0.255 scope global qr-08438cff-fc
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe61:ddb7/64 scope link 
       valid_lft forever preferred_lft forever
3: qg-f199827f-4f: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether fa:16:3e:1b:c3:d7 brd ff:ff:ff:ff:ff:ff
    inet 172.16.62.193/26 brd 172.16.62.255 scope global qg-f199827f-4f #router interface
       valid_lft forever preferred_lft forever
    inet 172.16.62.194/32 brd 172.16.62.194 scope global qg-f199827f-4f #instance floating ip addres
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe1b:c3d7/64 scope link 
       valid_lft forever preferred_lft forever

Routing table in namespace qrouter-xxxx

ubuntu@fair-stone:~$ sudo ip netns exec qrouter-a2e0f664-e969-4df1-a2df-16adfbe82cf3 ip r
default via 172.16.62.254 dev qg-f199827f-4f 
172.16.62.192/26 dev qg-f199827f-4f  proto kernel  scope link  src 172.16.62.193 
192.168.0.0/24 dev qr-08438cff-fc  proto kernel  scope link  src 192.168.0.1

Iptables in namespace qrouter-xxx

ubuntu@fair-stone:~$ sudo ip netns exec qrouter-a2e0f664-e969-4df1-a2df-16adfbe82cf3 iptables -t nat -L -nv
Chain PREROUTING (policy ACCEPT 1721 packets, 628K bytes)
 pkts bytes target     prot opt in     out     source               destination         
 1848  635K neutron-vpn-agen-PREROUTING  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain INPUT (policy ACCEPT 908 packets, 372K bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 8 packets, 542 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    8   542 neutron-vpn-agen-OUTPUT  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain POSTROUTING (policy ACCEPT 98 packets, 6014 bytes)
 pkts bytes target     prot opt in     out     source               destination         
  108  6773 neutron-vpn-agen-POSTROUTING  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
  104  6485 neutron-postrouting-bottom  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain neutron-postrouting-bottom (1 references)
 pkts bytes target     prot opt in     out     source               destination         
  104  6485 neutron-vpn-agen-snat  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain neutron-vpn-agen-OUTPUT (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DNAT       all  --  *      *       0.0.0.0/0            172.16.62.194        to:192.168.0.3

Chain neutron-vpn-agen-POSTROUTING (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    4   288 ACCEPT     all  --  !qg-f199827f-4f !qg-f199827f-4f  0.0.0.0/0            0.0.0.0/0            ! ctstate DNAT

Chain neutron-vpn-agen-PREROUTING (1 references)
 pkts bytes target     prot opt in     out     source               destination         
   15   900 REDIRECT   tcp  --  *      *       0.0.0.0/0            169.254.169.254      tcp dpt:80 redir ports 9697
   97  5940 DNAT       all  --  *      *       0.0.0.0/0            172.16.62.194        to:192.168.0.3

Chain neutron-vpn-agen-float-snat (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    6   471 SNAT       all  --  *      *       192.168.0.3          0.0.0.0/0            to:172.16.62.194

Chain neutron-vpn-agen-snat (1 references)
 pkts bytes target     prot opt in     out     source               destination         
  104  6485 neutron-vpn-agen-float-snat  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
    0     0 SNAT       all  --  *      *       192.168.0.0/24       0.0.0.0/0            to:172.16.62.193

IPtables nat rules in namespace qrouter-xxx

ubuntu@fair-stone:~$ sudo ip netns exec qrouter-a2e0f664-e969-4df1-a2df-16adfbe82cf3 iptables -S -t nat
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-N neutron-postrouting-bottom
-N neutron-vpn-agen-OUTPUT
-N neutron-vpn-agen-POSTROUTING
-N neutron-vpn-agen-PREROUTING
-N neutron-vpn-agen-float-snat
-N neutron-vpn-agen-snat
-A PREROUTING -j neutron-vpn-agen-PREROUTING
-A OUTPUT -j neutron-vpn-agen-OUTPUT
-A POSTROUTING -j neutron-vpn-agen-POSTROUTING
-A POSTROUTING -j neutron-postrouting-bottom
-A neutron-postrouting-bottom -j neutron-vpn-agen-snat
-A neutron-vpn-agen-OUTPUT -d 172.16.62.194/32 -j DNAT --to-destination 192.168.0.3
-A neutron-vpn-agen-POSTROUTING ! -i qg-f199827f-4f ! -o qg-f199827f-4f -m conntrack ! --ctstate DNAT -j ACCEPT
-A neutron-vpn-agen-PREROUTING -d 169.254.169.254/32 -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 9697
-A neutron-vpn-agen-PREROUTING -d 172.16.62.194/32 -j DNAT --to-destination 192.168.0.3
-A neutron-vpn-agen-float-snat -s 192.168.0.3/32 -j SNAT --to-source 172.16.62.194
-A neutron-vpn-agen-snat -j neutron-vpn-agen-float-snat
-A neutron-vpn-agen-snat -s 192.168.0.0/24 -j SNAT --to-source 172.16.62.193

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.