I have an Ubuntu 14 LTS server that has been in production for about 6 months.
Today I was making a routine tweak to the network configuration and everything went sideways. I have 4 vlan network interfaces that suddenly won't route traffic. I can't ping anything on any of those network interfaces.
I have rebooted the server several times and even restored to a backup from last night before I made any changes, but nothing will get these working.
At first I thought the problem may have been created by the datacenter network engineers, but I have added a couple Windows boxes to one of the vlans and they are communicating between each other fine, but can't ping the freeswitch on the vlan. UFW is disabled.
I have a eth0 which has two IP addresses and which continues to work fine:
auto eth0 iface eth0 inet static address 192.168.1.13 netmask 255.255.255.0 gateway 192.168.1.1 dns-nameservers 192.168.1.12 192.168.1.11 10.1.0.10 hwaddress ether 00:50:56:b1:23:b0 auto eth0:0 iface eth0:0 inet static address 192.168.1.14 netmask 255.255.255.0
I have 3 vlans which are point-to-point circuits coming in on dedicated interfaces ( I have obfuscated the public ip addresses ):
auto eth2 iface eth2 inet static address 10.2.0.12 netmask 255.255.255.0 hwaddress ether 00:50:56:b1:71:7a post-up route add -net 192.168.211.32 netmask 255.255.255.224 gw 10.2.0.1 auto eth3 iface eth3 inet static address 10.3.0.13 netmask 255.255.255.0 hwaddress ether 00:50:56:b1:1c:cd post-up route add -net 192.168.211.64 netmask 255.255.255.224 gw 10.3.0.1 auto eth4 iface eth4 inet static address 144.?.?.206 netmask 255.255.255.252 hwaddress ether 00:50:56:b1:3d:10 post-up route add -net 199.?.?.67 netmask 255.255.255.255 gw 144.?.?.205
There's a 4th vlan which is the one we tried to turn up today, I have removed the "post-up route add" because when I tried to add it, that's when things went sideways:
auto eth1 iface eth1 inet static address 10.100.0.11 netmask 255.255.255.0 hwaddress ether 00:50:56:b1:82:59
Here is the routing table ( I obfuscated the public ip address ):
Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 192.168.1.1 0.0.0.0 UG 0 0 0 eth0 10.2.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth2 10.3.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth3 144.?.?.204 0.0.0.0 255.255.255.252 U 0 0 0 eth4 172.16.43.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1 192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0 192.168.211.32 10.2.0.1 255.255.255.224 UG 0 0 0 eth2 192.168.211.64 10.3.0.1 255.255.255.224 UG 0 0 0 eth3 199.?.?.67 144.?.?.205 255.255.255.255 UGH 0 0 0 eth4
eth3 ( 10.3.0.13 ) has been in production the longest. I have added two Windows computers to that vlan for troubleshooting purposes as 10.3.0.99 and 10.3.0.98 and they can ping each other but neither can ping 10.3.0.13, and the ubuntu server just gets this when trying to ping one of them:
itas@FreeSWITCH2:~$ ping 10.3.0.99 PING 10.3.0.99 (10.3.0.99) 56(84) bytes of data. From 10.3.0.13 icmp_seq=1 Destination Host Unreachable From 10.3.0.13 icmp_seq=2 Destination Host Unreachable From 10.3.0.13 icmp_seq=3 Destination Host Unreachable
Based on that ping output, it appears to be selecting the right interface, because 10.3.0.13 is the ip on that interface, however "sudo tcpdump -i eth3" never shows a single packet in or out and "sudo tcpdump -i any icmp" shows this:
20:10:52.378575 IP 10.3.0.13 > 10.3.0.13: ICMP host 10.3.0.99 unreachable, length 92
This looks like the icmp rejection never touches the "wire" ( this is a vmware environment ). Wireshark on the windows boxes never sees the icmp echo requests and I have configured VMware to allow promiscuous. One of the Windows boxes is on the same host and the other test one is on a different host.
From everything I've read, this looks like it should be a problem with the routing table, but I can't seem to find a problem with the routing table, and I can't seem to convince this box to send a packet out on eth3 or any of the vlan interfaces.
I'm at a complete loss as to what's wrong with all these network interfaces.