Can You Improve Fault Tolerance with Multiple IP Interfaces on the Same Subnet?

Having two interfaces in a module on the same subnet does not give you two interfaces into and out of the module. While you do get two interfaces into the module, STCP will use just one interface when sending packets out of the module. It does this regardless of which interface a client has connected to.

For example, given these two interfaces on the 172.16.1.0/24 subnet

%phx_vos#sdlmux.m16.10.11-2        IP: 172.16.1.116                     MAC: 00:00:a8:42:52:22

%phx_vos#sdlmux.m16.11.11-2        IP: 172.16.1.226                     MAC: 00:00:a8:43:52:22

Notice in figure 1 that the packet sent to 172.16.1.226 uses the destination MAC address of 00:00:a8:43:52:22 but the reply comes from 00:00:a8:42:52:22, the MAC address for 172.16.1.116 even though in the packet the reply is shown as coming from 172.16.1.226.

Figure 1 – connection request to module (trace taken on client)

If the module initiates a connection instead of receiving one, it will always use the first (116) interface.

Figure 2 – connection request from module to another server (trace taken on server)

Even if you bind to the second (226) IP address when you send out the packet it leaves via the first (116) interface

Figure 3 – connection request from module to another server (trace taken on other server)

client application bound to the 172.16.1.226 IP interface

Only if the first (116) interface goes down will the module shift to using the second (226) interface. While connections to the second (226) interface will be maintained, connections to the first (116) interface will fail.

Interface 172.16.1.116 fails – note the change in source MAC address

Figure 4 – Changes in MAC when first interface fails (trace taken on client)

How can you tell which interface will be used to send traffic?

There are several ways.  The first is to use packet_monitor or some other protocol analyzer on the remote host or the network to evaluate the source Mac addresses, see figures 1 thru 4.

Second is to use netstat to look at the interface, generate some traffic, run netstat again and compare. Only one interface should have incremented its “transmitted frames” counter.

Figure 5 – Using netstat to compare transmitted frames counter

The third way is to connect to a service on another system and see what local source address is used. Unless you explicitly bind to another address the source IP address will correspond to the active interface.

Figure 6 – what interface is used for outgoing connections

It is my opinion that having multiple interfaces on the same subnet just confuses things and is usually not worth the confusion. If for some reason you need to IP addresses on the same subnet add an alias to the first interface.

ifconfig #sdlmux.m16.10.11-2 172.16.1.226 -add -alias

Adding IP address 172.16.1.226 to interface %phx_vos#sdlmux.m16.10.11-2

Figure 7 – adding an alias to an interface