It took my box a while, but it seems I finally hit a routing cache issue in 2.6.9. It seems the cache overflows.
Someone on LARTC tracked it down and posted a notice with his findings in December. A post that sounds similar recently appeared on LKML, too.
dmesg: dst cache overflow printk: 202 messages suppressed. dst cache overflow
rebecca:~# ping 192.168.0.2 PING 192.168.0.2 (192.168.0.2): 56 data bytes ping: sendto: No buffer space available ping: wrote 192.168.0.2 64 chars, ret=-1 64 bytes from 192.168.0.2: icmp_seq=3 ttl=64 time=0.3 ms ping: sendto: No buffer space available ping: wrote 192.168.0.2 64 chars, ret=-1
I increased my default router cache size by a factor of 2 and it’s back up and happy again. I should probably upgrade to a newer kernel with this bug fixed, but I am probably good for another 80 days. Have to protect that uptime!
rebecca:~# grep ip_dst_cache /proc/slabinfo ip_dst_cache 16545 16590 256 15 1 : tunables 120 60 0 : slabdata 1106 1106 0 rebecca:~# echo 32768 > /proc/sys/net/ipv4/route/max_size