Per RFC 3927 (emphasis mine):
To participate in wide-area IP networking, a host needs to be
configured with IP addresses for its interfaces, either manually by
the user or automatically from a source on the network such as a
Dynamic Host Configuration Protocol (DHCP) server. Unfortunately,
such address configuration information may not always be available.
It is therefore beneficial for a host to be able to depend on a
useful subset of IP networking functions even when no address
configuration is available. This document describes how a host may
automatically configure an interface with an IPv4 address within the
169.254/16 prefix that is valid for communication with other devices connected to the same physical (or logical) link.
So, to summarize, the 169.254/16 subnet was reserved to allow for communications between two physically (or logically, e.g. through a switch) connected devices without having to rely on DHCP or manual assignment of static IP addresses.
To quote the Wikipedia article on private networks,
If a host on an IEEE 802 (Ethernet) network cannot obtain a network
address via DHCP, an address from 169.254.1.0 to 169.254.254.255 may
be assigned pseudorandomly.
As an example of its usage, say you have two embedded devices that need to communicate with one another via a direct Ethernet connection, but you want one or both devices to first look for DHCP so they also have the possibility of being connected into a local area network. When the devices are connected directly together and there is no DHCP available, the devices may use a protocol such as multicast DNS to discover each other automatically, but each device must first have some arbitrary IP address assigned so they can actually talk to each other over the Ethernet link. Thus, the operating system automatically assigns an IP address to the Ethernet device from the 169.254/16 network.
Now, as for why that route is there in the routing table by default...if you look at
/etc/networks, you can see that the 'link-local' network is defined as being in this subnet:
$ cat /etc/networks
# symbolic names for networks, see networks(5) for more information
If you take this file's advice and check out
man networks, you'll see:
This file is read by the route(8) and netstat(8) utilities.
So the route is created automatically on boot after
route reads that file.
The route is only necessary if you plan to actually use link-local communications and don't want to assign a static IP address to the connected devices. But it doesn't do any harm to leave it in if you don't.
However, if you really want to remove it, you can do so with
sudo route del -net 169.254.0.0 gw 0.0.0.0
Or to remove it permanently, you should be able to just comment out its entry in
/etc/networks and reboot.