Super-Networking Blog

So as many of you know who have used it in the past netflow is a great tool. Netflow gives you detailed information about traffic flowing through your routers. You can find out what IPs the traffic is coming from and going to, you can see what protocols, what ports and how much traffic is going through that router. Big help if you are trying to find what a traffic spike is or why your Internet connection is maxed out.

Typically you need software to collect the exported netflow data and then compile it into some easy to read form. This software isn’t typically cheap or what if you need it now and don’t have time to install a collector. I will give you the commands you need to get a quick look at the traffic flowing right now.

First thing you have to do is have the router watch the flows:

Under each interface type the command “ip route-cache flow”

Exit the interface config and add the command “ip flow-export version 5” to select a version.

Let is collect traffic for a little bit, shouldn’t impact the performance or your router.

Then to see the traffic break down type the command “show ip cache flow”

This will give you the basic traffic breakdown going through you router. Look at Pkts column to see where the heavy hitters are. When you are down looking at it just leave it running on your interfaces, won’t affect performance and will be in place for when you have a netflow collector installed.

So I just started a job for a new company. One of the first things I did was look for easily resolvable issues on the network. Some symptoms to look for are any packet loss or high latency on a local LAN link, CRC errors on your switch ports and router ports, ultra slow RDP connections, or slow web surfing. These are just some places to start.

One issue I ran across at my new job that was causing slow web access was a duplex mismatch on our border router. One side was set to static 100/full and the other end was set to auto. You may thing that auto will detect the other end is at 100/full but this is not the case. If both ends are not running at auto they cannot negotiate the speed and duplex settings. In turn the auto end will randomly guess what speed and duplex to run at causing packet loss and slow downs.

To take this a step farther I want to share my experience with what to put these settings at. For point to point cables running at less than 1Gbps you should set both sides to static 100/full. For servers that are running 100Mbps NIC cards or 100Mbps switch ports you should set both sides to 100/Full. You could set both ends to auto but when the server is under stress it may negotiate down to 10/full causing major slowdowns. Client workstations and the switch ports they are plugged into should always be auto because you never know when you will plug a new device into it and you would have to set every new device to 100/full to work correctly. If you are running at Gbps speeds you have to leave both sides at auto, according to the Gbps standard you should never set to static 1000/Full. Most network adapters to not even allow you to. There is a setting on newer Gigabit drivers that allow for 1000/Auto which is really auto negotiate but it is weighted to 1Gbps.

Another thing that can cause slow Internet browsing on the network is failed or misconfigured DNS forwarders. So most people on a Microsoft AD domain use the domain controllers for DNS on the clients. Then you setup external DNS servers as forwarders on the domain controllers. What happens is when a client tries to go to www.google.com for example the client will put a DNS request to the domain controller who will not know it and send it on to the DNS forwarder. The DNS forwarder will then relay the correct information through the domain controller back to the client. If the DNS forwarders on the domain controllers are inaccessible, misconfigured, or having large packet loss your web experience will be slow or not work at all.

Another one I ran across lately was a server plugged into the local LAN that we returning a latency of 3-4ms per ping. This should never be above 1ms for any period of time unless that server is under an extreme load and even then it is rare. More likely there is another problem. I checked speed/duplex settings and they were fine, replaced the cat 5 cable, changed switchports none of which worked. I logged into the console and it was extremely slow, we are talking 10mins to login and even one you are logged in it took forever to do anything. I rebooting into dos mode with IP connectivity and the pings were still 3-4ms. It was an older box and I figured it was a hardware issue, I had a spare box of the same specs, moved the drives and when it came back up it was 10 times faster. Also the pings we less than 1ms.

Another one I found this past week was a server that was having problems talking to the domain. You could RDP to it but could not login because it couldn’t talk to the domain. I ran a ping against it and I noticed about every 3rd packet was dropping. I replaced the cable and everything worked again.

If you have run across any network troubleshooting situations lately comment on the blog entry for others to learn.