Super-Networking

I have been working on cleaning up some network issues lately on our Cisco 7609. One of the issues that was showing up is when you would do a show ip traffic command there were over 100 million “bad hop count” errors. This count continued to increase no matter the time of day or the amount of traffic load. Bad hop count occurs when a packet is discarded because its time-to-live (TTL) field was decremented to zero. In researching this problem the description of “bad hop count” I found that the main reason this problem occurs is a back door bridge. A back door bridge is a bridge between network segments that is bypassing the router and causing spanning-tree issues. Well I was unable to find a backdoor bridge on our network so I started to run Ethereal, which is a packet analyzer, on different network segments filtering for low ttls. What I found was a program that maintains web server state was broadcasting on one of the network segments with a ttl of 1. So when an of these packets hit the router interface the ttl was counted down to zero and the packet was discarded causing “bad hop count” to increase. This product was very chatty and was causing 100s of errors a second. So far we have just disabled the program and the errors have stopped. So if you run into this problem on your network you might want to look into running ethereal or some other sniffer on your network and look for packets with a ttl of 1. Product was ScaleOut StateServer - http://www.scaleoutsoftware.com/

WordpressTags: Cisco, Networking, Router, Software