Tech/Many NICs in the same subnet
I recently had the chance to work on a proof of concept at work and faced an interesting challenge: what does it take to be able to use more than one network interface connected to the same subnet? One might expect that once you've got the ip addresses from your local dhcp server all is good and you can start sending and receiving traffic. I've learned that's not the case.
In this article I want to take notes about what's necessary for such configuration to work. I haven't fully understood (yet) all the pieces involved so his page will end up also being a tracker for my learnings
Charts
Quick and dirty notes
Let's say our subnet is has address 10.55.0.0/18 and our nics these addresses:
- eth0: 10.55.36.33
- eth1: 10.55.8.255
- eth2: 10.55.38.187
Then the necessary commands would be:
# ==================================================================
# Block 1: tables creation
echo "10 eth0" >> /etc/iproute2/rt_tables
echo "11 eth1" >> /etc/iproute2/rt_tables
echo "12 eth2" >> /etc/iproute2/rt_tables
# ==================================================================
# Block 2: Add rules to tables
ip route add 10.55.0.0/18 dev eth0 src 10.55.36.33 table eth0
ip route add default via 10.55.0.1 dev eth0 table eth0
ip route add 10.55.0.0/18 dev eth1 src 10.55.8.255 table eth1
ip route add default via 10.55.0.1 dev eth1 table eth1
ip route add 10.55.0.0/18 dev eth2 src 10.55.38.187 table eth2
ip route add default via 10.55.0.1 dev eth2 table eth2
# ==================================================================
# Block 3: bind ip addresses to tables
ip rule add from 10.55.36.33/32 table eth0
ip rule add to 10.55.36.33/32 table eth0
ip rule add from 10.55.8.255/32 table eth1
ip rule add to 10.55.8.255/32 table eth1
ip rule add from 10.55.38.187/32 table eth2
ip rule add to 10.55.38.187/32 table eth2
Quick and dirty explanation
The block 1 creates three tables. Tables are sorted and consulted by (ascending, if I understood correctly?) order of priority. The priorities here are the 10, 11 and 12 numbers. Tables with low numbers comes first, tables with high priority numbers come later.
To give an idea, the default would be something like this:
manu@astrolabio:~$ ip rule show
0: from all lookup local
32766: from all lookup main
32767: from all lookup default
So adding tables with priorities 10/11/12 would mean that such tables are consulted after the local table and before the main and default tables.
Then we move to adding rules to tables (block 2)
(to be continued)
Links and references
- [ServerFault] Simple Multihomed Linux Server Issue
- https://lartc.org/howto/ - Linux Advanced Routing & Traffic Control: poorly aged, from a quick skim seems less good than it claims to be
- Linux Advanced Routing mini how-to from linuxhorizon.ro
- https://kindlund.wordpress.com/2007/11/19/configuring-multiple-default-routes-in-linux/
- Another person that had the same issue... Doesn't explain much more than other websites, but has some nice commentary and links
- http://www.policyrouting.org/
- in particular: http://www.policyrouting.org/PolicyRoutingBook/ ("Policy Routing With Linux by Matthew G. Marsh")
- http://www.policyrouting.org/iproute2-toc.html -- iproute2 docs
