What's the difference between tun/tap vs bridge+vnet vs macvtap? (For virtualization KVM)

Question

I've just found a lot of different ways to do KVM networking. But I'm stuck about what's the right way to do it. I discovered that openstack uses macvtap to do neutron networking. And it looks good.

But what's the difference and why to use each way.

Way 1 [ OLD? TUN/TAP ]

http://www.shakthimaan.com/installs/debian-tun-tap-setup.html

/--------\   /----\   /----\   /----\   /--------\
|Internet|---|eth0|---|br0 |---|tap0|---|Guest NIC
\--------/   \----/   \----/   \----/   \--------/

Deprecated, right?

Way 2 [ Bridge+Vnet ] <- That's what virt-manager does

http://www.linux-kvm.com/content/using-bridged-networking-virt-manager

Basicly you create a bridge interface with your physical interface inside and

auto br0
#iface br0 inet dhcp
iface br0 inet static
address 172.16.0.100
network 172.16.0.0
netmask 255.255.0.0
broadcast 172.16.255.255
gateway 172.16.0.1
   bridge_ports eth2
   bridge_stp off
    bridge_fd 0
    bridge_maxwait 0

And when you start a virtual machine from virt-manager a vnet interface is created and added to the bridge. At least until where I know. No tun/tap interface is needed.

It worked quite well for a long time but now with saucy I've found problems.

https://bugs.launchpad.net/ubuntu/+source/core-network/+bug/1255516

Why can you add a new vnet interface to the bridge without the TAP interface?

Way 3 [ MACVTAP ]

Last is macvtap interface.

http://virt.kernelnewbies.org/MacVTap

It copies the TUN/TAP software interface but it does in a better way. Don't know what way, but it seems to be better.

What's the advantage of macvtap over the second way?

What's better?

Any help on this?

Answer 1

These methods are doing fundamentally different things. To understand why, you need to understand the layered model of networking. For our purposes here, layers 1, 2 and 3 are important:

• Layer 1 is the physical layer - this specifies things like what cables you can use, what voltage / current patterns represent 1s and 0s on that cable, how the devices at each end of a cable negotiate what bit rate they operate at and so on.
• Layer 2 is the link layer - this specifies what language things at each end of a cable talk to each other. Ethernet devices at this layer have things like frames and MAC addresses.
• Layer 3 is the network layer - this specifies how devices use a direct layer 2 link to another device to reach a third device that they can't reach directly at layer 2. Devices at this layer have IP addresses and routing tables.

MACVLAN / MACVTAP

MACVLAN creates a virtual layer 2 or link layer device, with its own MAC address, which shares the layer 1 or physical layer with an existing device. The most obviously understandable case is where you have an Ethernet device plugged into a network and you create a MACVLAN device based on that Ethernet device; now you have two Ethernet "devices" with different MAC addresses but which both transmit their frames on the same cable. I'll talk about MACVTAP a bit further down.

MACVLAN interfaces can interact in several different ways with the existing Ethernet interface, in particular when a frame appears on one of the interfaces which is addresses to the other one:

• In private mode, the frame is thrown away; it is not possible for the two interfaces to communicate with each other, only with external devices.
• In vepa mode, the frame is sent over the physical layer like any other frame. If you have the device plugged into a switch that is clever enough to spot that the frame then needs to be send back down the same port it arrived on, then it will be received by the same physical layer that sent it and then layer 2 will use the MAC to send it to the intended network interface.
• In bridge mode, when a frame appears on one device, it is checked to see if it is intended for the other and if so, it is sent there without going through layer 1.
• There are also a couple of more obscure modes.

Note that MACVLAN interfaces have an important restriction: They're not capable of address learning. So you can't bridge a MACVLAN interface to a second physical device and expect to be able to reach that second physical device over the first one. This works with the original Ethernet interface but not with a MACVLAN interface attached to it.

TUN/TAP

A TAP interface is also a new virtual layer 2 device but with no layer 1 attached to it. Instead, a program can get a file descriptor representing the physical layer. It can then write raw Ethernet frame data into that file descriptor and the kernel will treat it like any other Ethernet packet it receives on a real physical interface.

The big thing about TAP interfaces is that the physical layer is in user mode; any bit of software with the appropriate permissions can generate Ethernet frames in any way it likes and shove them into something the kernel treats the same as a real physical interface. This makes them very useful for things like VPNs and tunnelling; you can write whatever sort of tunnelling software you like in user space and there is no need to meddle in kernel space to get the frames into the networking stack, you just create a TAP device and write the frames into its file descriptor.

TUN devices are just like TAP devices except they operate at layer 3 instead of layer 2 and the user mode software has to write raw IP packets into the file descriptor instead of raw Ethernet frames.

Going back to MACVTAP devices, these are sort of mix-up between MACVLAN and TAP interfaces. Like TAP interfaces, a user-mode program can get a file descriptor and write raw Ethernet frames into it. Like a MACVLAN interface, those frames are then sent over the physical layer of a real Ethernet device. This allows you to easily adapt software that's written to use TAP devices to use a MACVLAN device instead.

VNet

This is conceptually similar to TUN/TAP networking but has a more developed control plane (so the user-mode software using it can configure the interface more flexibly) and a more optimised data plane (so you can move data through the virtual network device more efficiently).

All of these do similar things but have slightly different capabilities. All of them can be used to connect a VM to an Ethernet network:

• A virtualisation product can take Ethernet frames from the guest and write them into the file descriptor for a TAP device. That TAP device can be assigned its own IP address by the host, or it can slaved to a bridge along with an Ethernet interface to share the host's IP address, or iptables can be configured to forward traffic on it using NAT.
• A virtualisation product can that Ethernet frames from the guest and write them into the file descriptor for a MACVTAP device; these then get transmitted directly on an Ethernet device's physical layer, effectively giving the VM a "real" Ethernet device (though note that it is possible to create MACVLAN / MACVTAP devices for other types of network interfaces such as bridges).
• A virtualisation product can connect a virtio driver in the guest to the virtio driver in the host for very efficient networking.

Answer 2

It really depends what exactly you want to achieve

• TAP/ TUN

Doesn't matter it's VM or physical machine. TUN brings you a tunneled network and TAP a device. In short, you go through a tunneled network to reach out another network.

For instance, when configuring an OpenVPN network, you'd be given 10.8.0.6 on your client. VPN server 10.8.0.1 routes your request to another network (eg 192.168.x.x) behind. When using TAP, you'd receive an IP (192.168.10.10/24) directly from your target network (192.168.10.x/24). Simple.

• Bridge

"Linux Bridge" bridges VNET (from VM) to physical ethernet. If you want a VM (KVM based), bridge is a must between vnet and ethernet on host

Answer 3

I'd say it depends on your use case.

Automated adds/deletes of virtual hosts?

Give macvtap a try. Should also be performanter than macvlan (which is is roughly like adding another MAC to a physical device, infos arriving there are handled by the networking stack) or an additional bridge, as macvtap bypasses the network stack somehow and exports a tap character device directly. But don't nail me down on that. Besides both (macvlan/macvtap) share the same available modes (VEPA/hairpin, bridging, private), hairpinning only works if your switch supports reflective relay mode. (Packets arriving at the physical switch at port x must be allowed to leave the switch again on the same port x.)

Because the eth0 (or whichever one you use) is put into promiscuous mode when using a bridge, macvXXX modes are said to have higher throughputs.

The modes also define the 'amount' of isolation (can vhosts see each others traffic? how about the the hv?). How this works under the hood I do not yet know.

veth's (virtual ethernet pairs) are somewhat similar for isolation, you define two virtual interfaces, one gets attached to a bridge, the other to your VM. There the isolation is done by putting the vm-interface into it's own namespace so the devices are somewhat isolated. All traffic comes together at the bridge, but one vhost cannot see another one's vNICs.

In case you work with a bridge, you have additional configuration to do, and when the bridge is down, so are all your connections. When bringing the bridge back up, you may have to reconnect all virtual interfaces to the bridge again (or just reboot the complete hv...).

Bottom line: If you don't change your topology often, just go with bridging as you find the most information on it online, which is better than reading kernel code. Heck, even the iproute2-doc package itself lacks most of the information iproute2 actually has, even when you run bleeding-edge versions. Try finding out about man ip-tcp_metrics from the available manpages or ip-crefs.ps ...