How do I bind each of my server's NIC's to a separate IP?

Question

My question is "How do I bind each of my server's NIC's to a separate IP?"

My Ubuntu 12.04 server has two NIC's - both are attached to my home lan.

The NIC's have device names that were set in /etc/udev/rules.d/70-persistent-net.rules. I have called the lan0 and lan1. (see below for my 70-persistent-net.rules).

Each NIC acquires correctly its IP settings via DHCP from my router. The assignment of IP is done based on the MAC of the dhcp request. lan0 is assigned ip 172.23.80.1 whilst lan1 is assigned 172.23.80.2. (see below for ifconfig output and my /etc/network/interfaces).

My problem is that when I connect via ssh from a separate computer I always receive the connection from the same server MAC irrespective of the ip that I used in the ssh call.

Let me describe the following experiment that shows my problem. I use three terminals (term1, term2 and term3) on my Ubuntu desktop computer

term1:

arp -n
Address                  HWtype  HWaddress           Flags Mask            Iface
172.16.0.1               ether   00:50:7f:9d:2d:30   C                     wlan0
172.23.80.1                      (incomplete)                              wlan0

term2:

ssh me@172.23.80.1
me@172.23.80.1's password: 
Welcome to Ubuntu 12.04.1 LTS (GNU/Linux 3.2.0-29-generic-pae i686)

* Documentation:  https://help.ubuntu.com/
Last login: Thu Aug 16 12:37:48 2012 from 172.23.128.11
me@FERMI:~$ 

term1:

arp -n
Address                  HWtype  HWaddress           Flags Mask            Iface
172.16.0.1               ether   00:50:7f:9d:2d:30   C                     wlan0
172.23.80.1              ether   00:0d:61:22:3d:d6   C                     wlan0

term3:

ssh me@172.23.80.2
me@172.23.80.2's password: 
Welcome to Ubuntu 12.04.1 LTS (GNU/Linux 3.2.0-29-generic-pae i686)

* Documentation:  https://help.ubuntu.com/
Last login: Thu Aug 16 13:37:12 2012 from 172.23.128.11
me@FERMI:~$ 

term1:

arp -n
Address                  HWtype  HWaddress           Flags Mask            Iface
172.23.80.2              ether   00:0d:61:22:3d:d6   C                     wlan0
172.16.0.1               ether   00:50:7f:9d:2d:30   C                     wlan0
172.23.80.1              ether   00:0d:61:22:3d:d6   C                     wlan0

As you can see from this last arp both ssh connections are being serviced by a single NIC on the server even though the two ip's have been assigned to different NICs. This is my problem.

Here is my Server configuration:

me@FERMI:~$ ifconfig
lan0      Link encap:Ethernet  HWaddr 00:0d:61:22:3d:d6  
          inet addr:172.23.80.1  Bcast:172.31.255.255  Mask:255.240.0.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1452 errors:0 dropped:0 overruns:0 frame:0
          TX packets:496 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:113691 (113.6 KB)  TX bytes:72934 (72.9 KB)

lan1      Link encap:Ethernet  HWaddr 00:19:5b:68:93:57  
          inet addr:172.23.80.2  Bcast:172.31.255.255  Mask:255.240.0.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1103 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:85747 (85.7 KB)  TX bytes:684 (684.0 B)
          Interrupt:22 Base address:0x9000 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:14 errors:0 dropped:0 overruns:0 frame:0
          TX packets:14 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:700 (700.0 B)  TX bytes:700 (700.0 B)


me@FERMI:~$ cat /etc/network/interfaces 
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto lan0 
iface lan0 inet dhcp

auto lan1 
iface lan1 inet dhcp


me@FERMI:~$ cat /etc/udev/rules.d/70-persistent-net.rules 
# This file was automatically generated by the /lib/udev/write_net_rules
# program, run by the persistent-net-generator.rules rules file.
#
# You can modify it, as long as you keep each rule on a single
# line, and change only the value of the NAME= key.

# PCI device 0x1186:/sys/devices/pci0000:00/0000:00:1e.0/0000:03:02.0 (sundance)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:19:5b:68:93:57", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="lan1"

# PCI device 0x8086:/sys/devices/pci0000:00/0000:00:03.0/0000:02:01.0 (e1000)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:0d:61:22:3d:d6", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="lan0"


me@FERMI:~$ route 
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         OHM.UNIVERSE    0.0.0.0         UG    100    0        0 lan0
172.16.0.0      *               255.240.0.0     U     0      0        0 lan0
172.16.0.0      *               255.240.0.0     U     0      0        0 lan1


me@FERMI:~$ cat /proc/sys/net/ipv4/ip_forward
0


me@FERMI:~$ arp -n
Address                  HWtype  HWaddress           Flags Mask            Iface
172.23.128.11            ether   00:18:de:02:f0:5c   C                     lan0
172.16.0.1               ether   00:50:7f:9d:2d:30   C                     lan0


me@FERMI:~$ sudo ufw status
Status: inactive

Answer 1

Your NIC's are bound to separate IP's. The problem is the default route. The output of route shows that default routing goes via the lan0 NIC.

To resolve your claim just remove the default route with ip route del default or route del default.

However after that, you wont be able to route to outside your network. If you want to, then it's helpful to create two subnets for example 172.23.80.0 and 172.23.81.0 and route separately with via ip route 172.23.80.0/24 via 172.23.80.1 dev lan0 and ip route 172.23.81.0/24 via 172.23.81.1 dev lan1. For example

# /etc/network/interfaces
# The primary network interface
auto lan1
iface lan1 inet dhcp
post-up route del default
post-up ip route 172.23.81.0/24 via 172.23.81.1 dev lan1

Also possible but untested

route del default
ip route add 172.23.80.1/32 via 172.23.80.1 dev lan0
ip route add 172.23.80.2/32 via 172.23.80.2 dev lan1

Answer 2

You could use iptables to forcibly change the outgoing MAC/IP addresses for packets to the connecting computer.

Answer 3

So, I'd like to preface this by saying my iptables skills are not solid. But I believe @chronospoon is correct about the deficiency of @bersch's answer, even though it looks almost there.

Anyway, I search a bit and found LARTC (Linux Advanced Routing and Traffic Control). There is a howto on split access that looks like it explains exactly what you want. I'll quote:

The first is how to route answers to packets coming in over a particular provider, say Provider 1, back out again over that same provider.

Let us first set some symbolical names. Let $IF1 be the name of the first interface (if1 in the picture above) and $IF2 the name of the second interface. Then let $IP1 be the IP address associated with $IF1 and $IP2 the IP address associated with $IF2. Next, let $P1 be the IP address of the gateway at Provider 1, and $P2 the IP address of the gateway at provider 2. Finally, let $P1_NET be the IP network $P1 is in, and $P2_NET the IP network $P2 is in.

One creates two additional routing tables, say T1 and T2. These are added in /etc/iproute2/rt_tables. Then you set up routing in these tables as follows:

  ip route add $P1_NET dev $IF1 src $IP1 table T1
  ip route add default via $P1 table T1
  ip route add $P2_NET dev $IF2 src $IP2 table T2
  ip route add default via $P2 table T2

Nothing spectacular, just build a route to the gateway and build a default route via that gateway, as you would do in the case of a single upstream provider, but put the routes in a separate table per provider. Note that the network route suffices, as it tells you how to find any host in that network, which includes the gateway, as specified above. Next you set up the main routing table. It is a good idea to route things to the direct neighbour through the interface connected to that neighbour. Note the `src' arguments, they make sure the right outgoing IP address is chosen.

    ip route add $P1_NET dev $IF1 src $IP1
    ip route add $P2_NET dev $IF2 src $IP2

Then, your preference for default route:

    ip route add default via $P1

Next, you set up the routing rules. These actually choose what routing table to route with. You want to make sure that you route out a given interface if you already have the corresponding source address:

    ip rule add from $IP1 table T1
    ip rule add from $IP2 table T2

This set of commands makes sure all answers to traffic coming in on a particular interface get answered from that interface.

Warning

Reader Rod Roark notes: 'If $P0_NET is the local network and $IF0 is its interface, the following additional entries are desirable:

    ip route add $P0_NET     dev $IF0 table T1
    ip route add $P2_NET     dev $IF2 table T1
    ip route add 127.0.0.0/8 dev lo   table T1
    ip route add $P0_NET     dev $IF0 table T2
    ip route add $P1_NET     dev $IF1 table T2
    ip route add 127.0.0.0/8 dev lo   table T2

'
Now, this is just the very basic setup. It will work for all processes running on the router itself, and for the local network, if it is masqueraded. If it is not, then you either have IP space from both providers or you are going to want to masquerade to one of the two providers. In both cases you will want to add rules selecting which provider to route out from based on the IP address of the machine in the local network.