Neighbour table overflow

Under SecurePlatform you can sometimes see the following message in /var/log/messages

Jan 15 13:44:08 fw1 kernel: Neighbour table overflow.

This refers to the ARP cache a.k.a. Neighbour table.

If you’re running a gateway with lot’s of interfaces or big subnets, you might see many nodes over Layer-2, so communication to them fills your ARP table and sometimes overflows it, which can lead to connectivity errors.

The ARP cache table has a maximum size, which can be displayed withcat /proc/sys/net/ipv4/neigh/default/gc_thresh3.
You can verify the actual amount of ARP entries either with arp -an | wc -l or with ip neighbor show |wc -l. Proxy ARP entries are only displayed when using the arp command.

Periodically and automatically the entries in the ARP cache are verified. At a specified interval, a garbage collector is running and removes entries that are no longer used. The interval can be verified with cat /proc/sys/net/ipv4/neigh/default/gc_interval, by default it’s 30 seconds.

The garbage collector is controlled by three variables:
gc_thresh1, which is the minimum number of entries in the ARP cache. If the actual number of entries are below this value, the garbage collector will not run.

gc_thresh2, which is the soft maximum number of entries. If the actual number of entries is above this value for more than 5 seconds, the garbage collector will run.

gc_thresh3, which is the hard maximum number of entries. If the actual number of entries is above this value, the garbage collector with immediately run.

gc_thresh3 is also the maximum value of ARP entries that can be kept in the table.

The default values are quite low, so you might want to increase them.

You can do this on the fly with the following CLI commands:

sysctl -w net.ipv4.neigh.default.gc_thresh3=4096
sysctl -w net.ipv4.neigh.default.gc_thresh2=2048
sysctl -w net.ipv4.neigh.default.gc_thresh1=1024

This does not survice a reboot.

To survive a reboot, add this lines in the /etc/sysctl.conf file

net.ipv4.neigh.default.gc_thresh3 = 4096
net.ipv4.neigh.default.gc_thresh2 = 2048
net.ipv4.neigh.default.gc_thresh1 = 1024

Afterwards run the command sysctl -p for the changes to take effect and then reboot.