Super-Networking Blog

So as many of you know who have used it in the past netflow is a great tool. Netflow gives you detailed information about traffic flowing through your routers. You can find out what IPs the traffic is coming from and going to, you can see what protocols, what ports and how much traffic is going through that router. Big help if you are trying to find what a traffic spike is or why your Internet connection is maxed out.

Typically you need software to collect the exported netflow data and then compile it into some easy to read form. This software isn’t typically cheap or what if you need it now and don’t have time to install a collector. I will give you the commands you need to get a quick look at the traffic flowing right now.

First thing you have to do is have the router watch the flows:

Under each interface type the command “ip route-cache flow”

Exit the interface config and add the command “ip flow-export version 5” to select a version.

Let is collect traffic for a little bit, shouldn’t impact the performance or your router.

Then to see the traffic break down type the command “show ip cache flow”

This will give you the basic traffic breakdown going through you router. Look at Pkts column to see where the heavy hitters are. When you are down looking at it just leave it running on your interfaces, won’t affect performance and will be in place for when you have a netflow collector installed.

So if you are multi-homed on BGP and you want to be able to decide yourself which path incoming and outgoing your traffic will take you will need to make a few changes. By default BGP will try to make the best decision on what path to take. Problem is a lot of time it will just choose the lowest AS number which might not be the best route.

router bgp 11111
neighbor 2.2.2.2 weight 500
neighbor 3.3.3.3 weight 400
neighbor 3.3.3.3 route-map prepend out

ip prefix-list aggregate seq 5 permit 1.1.1.0/24

route-map prepend permit 10
match ip address prefix-list aggregate
set as-path prepend 11111 11111 11111 11111 11111

Under your BGP config on your router add some weight statements, the higher the weight the more preferred route for outgoing traffic. The route-map prepend out statement you will want to put on the neighbor that you don’t want traffic coming in on. Incoming BGP traffic will always try to take the smallest AS number first so if you prepend your AS number onto your neighbors AS path it will seem like the worst path. This will give you the effect of incoming traffic on the neighbor without the prepend router-map.

So like most IT security minded people I want to use SSH on everything I can because with telnet your username and password are sent over the wire in clear text. Now I know most of you say well if you are internal what matters it your password is sent in clear text who would be listening? Well a lot of people could be, your boss, your fellow employee in IT, a rouge server, etc…

One thing that you can do to mitigate the threat to to make sure you are switching all your data since a switch doesn’t broadcast most data like hubs do. That still leaves it open to people who have access to the network switches and can sniff your port or the port you are going to. So you can use SSH since it is encrypted and that will protect you. Now not everything supports SSH, both on the software side and on the hardware side. I can’t help with the hardware, if your switch or router doesn’t support SSH either you have to buy a new IOS for it or stick to telnet at your own risk.

As for the software side if you have a program that needs to connect to your network devices over telnet and it can’t be switched to SSH and you really need it listen up.

You can use the access-class command under line vty 0 4 to lockdown what IPs have access to SSH or Telnet.

Example Commands:

ip access-list extended vty
permit tcp host 10.10.0.5 any eq telnet log
permit tcp any any eq 22 log
deny tcp any any log

line vty 0 4

access-class vty in

transport input telnet ssh

So that ACL would allow 10.10.0.5 to telnet to the network device and anyone to SSH. FYI that older versions of the IOS don’t allow you to use extended ACLs so you would only be able to determine IPs not ports/services.

If you are like me you like to take a running-config dump it to a text file open it in a text editor before making any major changes. You can then plan out all the config changes you want to make and save it to a separate file so you have the current config and the future config saved.

When you want to make the changes to your network device you can just cut and paste in the parts that are changing instead of having to type it all in and possibly commit typos.

One of the annoyances I have run into is when you have a large config there are a ton of pauses in the config text file that show up as –More–. These are really annoying and can mess up your cut and paste config changes if you miss one.

To fix this issue before you type “show run” type “term len 0″. This will cause the config to display without pausing and if you have logging turned on in putty if will dump to text without the annoying –More–s in it. After you save your file you are going to want to type “term len 24″ to get it to pause again while you are making the changes.

So ICMP is a double edge sword, it is great for helping you troubleshoot network issues and verifying the most basic connectivity is working. It also can give away a lot of information about what is going on in your network to people who shouldn’t know.

The question then becomes what kind of ICMP types do you let through access-lists that you have on your border routers. You want to be able to ping outside hosts and recieve the replys, you might want people to be able to ping your public facing servers, you are going to want to be able to traceroute outside of your network.

Here is my recommendation for your access-list:

deny icmp any any fragments
permit icmp any any echo
permit icmp any any echo-reply
permit icmp any any traceroute
permit icmp any any source-quench
permit icmp any any packet-too-big
permit icmp any any time-exceeded

*Note this is just a section to the access-list not the entire access-list.

If you are thinking that the permit icmp any any traceroute will allow your traceroute from your windows box to function you will find that not the case. For Windows tracert you need to allow that permit icmp any any time-exceeded.

Had some routers automatically change and some routers not change there time this weekend so I put in a workaround to get them all to play nice with this DST change. These commands are for IOS.
Remove the line “clock summer-time CST recurring” from your config. (You might have to change the timezone in your command)

Add the following line “clock summer-time CST date Mar 11 2007 2:00 Nov 4 2007 2:00” this manually tells the router when to adjust for DST.

The time difference typically doesn’t affect the routers functionality but will cause syslog timestamp issues.

I have been working on getting netflow working for all traffic going through our 6500 and 7600 series Cisco devices. We are running Supervisor 720/PFC3a blades, native IOS.
Commands added to get Netflow working:

mls aging fast time 8 threshold 127

mls aging normal 32

mls nde sender version 5

ip flow-export destination

ip flow-export version 5

ip flow-export source

Under VLAN interfaces and WAN port interfaces:

ip route-cache flow

With these command I was able to get some traffic but not all. One thing I noticed was that I had “no mls flow ip” in my config. This command controls the flow mask that decides what in the netflow traffic to track.

When I tried to change this to “mls flow ip full” or just about any other combination of this command I would get the following errror:

% Unable to configure flow mask for ip protocol: full. Reset to the default flow mask type: none

After a ton of digging and some help from Cisco TAC we are able to get around this with the command “mls flow ip interface-destination-source“. The reason this was happening was because we are doing NAT on our MSFC. Nat uses a flow mask as well and with the exception of the above setting will conflict with mls mask settings.

After changing the flow mask everything is now coming through.

Recently I was working on adding a new /24 network into our BGP advertisement networks. I added the new network into the network list under the BGP config, I added it into my prefix-list and route-map that restricts what networks we advertise to our BGP peers.

When I checked what routes were being advertised using the “show ip bgp neighbors X.X.X.X advertised-routes” command the new network was not there. If I ran the command “show ip bgp all” the network did not show up either.

When I ran the command “ip bgp route-map localonly” which points to “ip as-path access-list 10 permit ^$” the new network also was not listed.

I tried clearing the BGP sessions, removing and readding the route-map and prefix-list but it did not help.

I had added a ip route X.X.X.X 255.255.255.0 Null0 into my routing table so I knew that wasn’t the problem. I ran the command “ip bgp prefix-list” and saw that the new network was not listed there either. I ran the command “ip prefix-list” and the new network was there.

What ended up being the problem was that if the network you are adding matches the ip class you are adding you do not need a mask but if it doesn’t you need to add the mask in the BGP config.

Example (Not Actual IPs):

If you add a 9.X.X.X network to the BGP config it is a class A so you either would have to advertise “network 9.0.0.0″ or if say yours was 9.1.1.0 then you would have to advertise “network 9.1.1.0 mask 255.255.255.0″ for it to work. If you add a network like 200.1.1.0 since this is in the Class C you could just add “network 200.1.1.0″ into your BGP config.

Here is a good troubleshooting doc on this:

BGP No Advertise

Tries to go through the maintenance of upgrading our CSM and the IOS on our Sup 720s this past week and ran into some problems. The CSM upgrade went well but after rebooting the CSM with the new software one of our Sup 720s burned out. It was one of those freak things but now working on getting the Supervisor replaced and will be finishing the IOS upgrade then. I had a couple of documents in this post to help you with your upgrades.

The following is some good info for upgrading your CSM when you have a Sup720 instead of a Sup 1 or 2.

CSM> upgrade 127.0.0.zz c6slb-apc.revision-num.bin

The zz is 12 if the supervisor engine is installed in chassis slot 1.

The zz is 22 if the supervisor engine is installed in chassis slot 2.

Full Doc 

When you add a static nat in your Cisco router for one of your servers the router then knows the internal and external IP. When you do a DNS query on that server to an outside DNS server the router will translate your returned address to the internal IP for you. See below for the official explanation:

Q. Does Cisco IOS NAT support DNS queries?

A. Yes, Cisco IOS NAT does translate the address(es) which appear in DNS responses to name lookups (A queries) and inverse lookups (PTR queries). If an outside host sends a name-lookup to a DNS server on the inside, and that server responds with a local address, the NAT code translates that local address to a global address. The opposite is also true, and is how Cisco supports IP addresses that overlap. An inside host queries an outside DNS server, the response contains an address that matches the ACL specified on the outside source command, and the code translates the outside global address to an outside local address.

Time-to-live (TTL) values on all DNS resource records (RRs) which receive address translations in RR payloads are automatically set to zero.

Cisco IOS NAT does not translate IP addresses embedded in DNS zone transfers.