Changing of the Guard – Replacing a firewall and gratuitous ARP

Replacing a firewall, especially with dissimilar equipment, can be a long and painstaking task, trying to match up rules, filters and such. But all of that is often in the control of the local network administrator. However, there often seems to be a very important part of such a change that is forgotten, and in some cases isn’t in their control.

By way of an overly-simplified background explanation: For devices to communicate over Ethernet, the packets are addressed to the hardware or Media Access Control (MAC) address of the device it is being sent to on the local network. So, if Computer A wants to send a packet to Computer B, it puts the MAC address of Computer B in the packet. Computer B reads all the packets it receives, and only processes those that bear its MAC address (or broadcast packets). This goes back to the days of shared networks where all the network packets were visible to all the devices on the network. Nowadays with modern switched networks devices only see the packets meant for it, as the switch knows which MAC addresses are generally on which ports and direct the packets only on the appropriate port.

If Computer A doesn’t know the MAC address of Computer B, it sends out a broadcast packet (“ARP Request”), asking for any device that has a specific IP address, to respond with its MAC address.  Computer B sees the request, identifies that the request is for its own IP, and responds (“ARP Reply”) Computer A then caches that info and can now direct the data directly to Computer B. All of this is part of what is known as the “Address Resolution Protocol”, or ARP.

The network switches and some other devices like routers can also cache this information as it is learned, so that it directs packets only on the appropriate ports. And this is where the problem comes in. When replacing equipment that will have the same IP address, but a different MAC address, the network devices need to re-learn the MAC address. Typically if they don’t receive a response to a directed packet, they will send out another broadcast packet asking for a new MAC address. But some network devices don’t always handle this well – especially when what is being replaced is a network device like a firewall. The usual solution is to “flush the ARP cache” – in other words, make the affected devices forget all about what is connected where and relearn everything. This can take a while if you have a lot of devices. And you need to have administrative access to the device to flush the cache.

An alternative to this is for the replacement device to send out an ARP Reply packet as if it received a request, but instead send it as a broadcast. When all the other devices received the broadcast ARP packet, they updated their local caches. This packet is referred to as a “Gratuitous ARP”.

Some devices do this automatically – for instance, some models of firewalls in highly-available configurations will send a gratuitous ARP when the firewall is failed over.

Where it can be a problem is in a situation where you don’t have access to the upstream router, and you cannot coax the device into sending a gratuitous ARP – for instance, many small office situations with a non-redundant firewall and Internet service coming from a provider such a Verizon FiOS, which does not have a provider-supplied cable modem you have access to like with most cable providers.

We recently ran into this situation at a client’s office. The firewall was being replaced – it was originally a Linksys router you would typically use at home instead of a heavy-use office, and was being replaced with a Cisco PIX – an old piece of equipment but still more than capable for the needs of the office. They used Verizon FiOS for their provider, so no accessible upstream router (even the Verizon ONT was located elsewhere).

Everything was set up properly on the PIX, and when we cut over to it – nothing. No packets seemed to come in or go out. All sorts of debugging and rebooting of the PIX did nothing to help. Verizon support was also not helpful – they said it could be several hours before the ARP cache would flush, and about as long to get someone to do it manually. And there didn’t seem to be a way to get the PIX to send the gratuitous ARP. Reissuing the commands to set up the interface would supposedly do it, but had no effect.

So, to get around this problem, we turned to Linux.

Most Linux distributions have a command available called “arping”. The command is sort of an “ARP Ping”, which can send an ARP Request and wait for the reply. It also has the ability to send a gratuitous ARP. Of course, it would do so with the MAC address of the interface used, but what we want it to do is to send the ARP out with the MAC address of the firewall. Fortunately we can do that as well.

We had a spare interface on a server nearby (a laptop running Linux or booting a live Linux image that had arping available would work just as well), so we connected the interface to the upstream router in place of the firewall. We then manually brought up the interface with the IP address of the firewall, but we specified the MAC of the firewall’s outside interface in the ifconfig command, like so:

ifconfig eth1 hw ether 00:11:22:33:44:55 address 192.168.1.1 netmask 255.255.255.0

This tells the interface to accept packets with the MAC address of 00:11:22:33:44:55, and if we see an ARP Request on this interface for our IP, we’ll respond with this MAC as well. Then, we need to tell the upstream router about it, so we issue the arping:

arping –U –c 5 –I eth1 192.168.1.1

You use the IP address of the device we are spoofing and that we assigned in the ifconfig command. The “-U” tells it to send an unsolicited (gratuitous) ARP, and the “-c 5” just says to send it 5 times – this seems to allow some slower devices to still pick up on the change. Once we did this, a tcpdump showed that we suddenly were receiving a ton of traffic that wasn’t there before. We disconnected from the Linux box and connected the PIX - and everything was working as expected.

So keep in mind – if you are changing out a firewall and don’t have control of the network devices on either side of it, think about how you’d deal with the ARP cache issue – and perhaps have a Linux box on standby.

Related Content: 

• Human Firewalls: What They Are and Why They’re Important
• Firewalls 101: The Basics
• Insider Threats: When Your Firewall Security Can’t Protect You