1

I have an Ubuntu 14 LTS server that has been in production for about 6 months.

Today I was making a routine tweak to the network configuration and everything went sideways. I have 4 vlan network interfaces that suddenly won't route traffic. I can't ping anything on any of those network interfaces.

I have rebooted the server several times and even restored to a backup from last night before I made any changes, but nothing will get these working.

At first I thought the problem may have been created by the datacenter network engineers, but I have added a couple Windows boxes to one of the vlans and they are communicating between each other fine, but can't ping the freeswitch on the vlan. UFW is disabled.

I have a eth0 which has two IP addresses and which continues to work fine:

auto eth0
iface eth0 inet static
address 192.168.1.13
netmask 255.255.255.0
gateway 192.168.1.1
dns-nameservers 192.168.1.12 192.168.1.11 10.1.0.10
hwaddress ether 00:50:56:b1:23:b0

auto eth0:0
iface eth0:0 inet static
address 192.168.1.14
netmask 255.255.255.0

I have 3 vlans which are point-to-point circuits coming in on dedicated interfaces ( I have obfuscated the public ip addresses ):

auto eth2
iface eth2 inet static
address 10.2.0.12
netmask 255.255.255.0
hwaddress ether 00:50:56:b1:71:7a
post-up route add -net 192.168.211.32 netmask 255.255.255.224 gw 10.2.0.1

auto eth3
iface eth3 inet static
address 10.3.0.13
netmask 255.255.255.0
hwaddress ether 00:50:56:b1:1c:cd
post-up route add -net 192.168.211.64 netmask 255.255.255.224 gw 10.3.0.1

auto eth4
iface eth4 inet static
address 144.?.?.206
netmask 255.255.255.252
hwaddress ether 00:50:56:b1:3d:10
post-up route add -net 199.?.?.67 netmask 255.255.255.255 gw 144.?.?.205

There's a 4th vlan which is the one we tried to turn up today, I have removed the "post-up route add" because when I tried to add it, that's when things went sideways:

auto eth1
iface eth1 inet static
address 10.100.0.11
netmask 255.255.255.0
hwaddress ether 00:50:56:b1:82:59

Here is the routing table ( I obfuscated the public ip address ):

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.1.1     0.0.0.0         UG    0      0        0 eth0
10.2.0.0        0.0.0.0         255.255.255.0   U     0      0        0 eth2
10.3.0.0        0.0.0.0         255.255.255.0   U     0      0        0 eth3
144.?.?.204     0.0.0.0         255.255.255.252 U     0      0        0 eth4
172.16.43.0     0.0.0.0         255.255.255.0   U     0      0        0 eth1
192.168.1.0     0.0.0.0         255.255.255.0   U     0      0        0 eth0
192.168.211.32  10.2.0.1        255.255.255.224 UG    0      0        0 eth2
192.168.211.64  10.3.0.1        255.255.255.224 UG    0      0        0 eth3
199.?.?.67      144.?.?.205     255.255.255.255 UGH   0      0        0 eth4

eth3 ( 10.3.0.13 ) has been in production the longest. I have added two Windows computers to that vlan for troubleshooting purposes as 10.3.0.99 and 10.3.0.98 and they can ping each other but neither can ping 10.3.0.13, and the ubuntu server just gets this when trying to ping one of them:

itas@FreeSWITCH2:~$ ping 10.3.0.99
PING 10.3.0.99 (10.3.0.99) 56(84) bytes of data.
From 10.3.0.13 icmp_seq=1 Destination Host Unreachable
From 10.3.0.13 icmp_seq=2 Destination Host Unreachable
From 10.3.0.13 icmp_seq=3 Destination Host Unreachable

Based on that ping output, it appears to be selecting the right interface, because 10.3.0.13 is the ip on that interface, however "sudo tcpdump -i eth3" never shows a single packet in or out and "sudo tcpdump -i any icmp" shows this:

20:10:52.378575 IP 10.3.0.13 > 10.3.0.13: ICMP host 10.3.0.99 unreachable, length 92

This looks like the icmp rejection never touches the "wire" ( this is a vmware environment ). Wireshark on the windows boxes never sees the icmp echo requests and I have configured VMware to allow promiscuous. One of the Windows boxes is on the same host and the other test one is on a different host.

From everything I've read, this looks like it should be a problem with the routing table, but I can't seem to find a problem with the routing table, and I can't seem to convince this box to send a packet out on eth3 or any of the vlan interfaces.

I'm at a complete loss as to what's wrong with all these network interfaces.

  • What does ifconfig -a eth3 show do you see any IPs on eth3 from arp -a? Can you ping 10.3.0.13? Have you checked your VM settings? Are those interfaces in bridged mode? – meccooll Dec 2 '16 at 19:58
  • @meccooll I found the problem. Somehow my network adapters got shuffled and weren't in the same order any more. I thought the hwaddress field in /etc/network/interfaces allowed me to pin an interface to a nic but instead it allows me to overwrite the mac address. I have switched to a single nic using 8021q in ubuntu directly to avoid something like that from being able to happen again. – royce3 Dec 6 '16 at 12:18

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.