Super-Networking Blog

When you add a static nat in your Cisco router for one of your servers the router then knows the internal and external IP. When you do a DNS query on that server to an outside DNS server the router will translate your returned address to the internal IP for you. See below for the official explanation:

Q. Does Cisco IOS NAT support DNS queries?

A. Yes, Cisco IOS NAT does translate the address(es) which appear in DNS responses to name lookups (A queries) and inverse lookups (PTR queries). If an outside host sends a name-lookup to a DNS server on the inside, and that server responds with a local address, the NAT code translates that local address to a global address. The opposite is also true, and is how Cisco supports IP addresses that overlap. An inside host queries an outside DNS server, the response contains an address that matches the ACL specified on the outside source command, and the code translates the outside global address to an outside local address.

Time-to-live (TTL) values on all DNS resource records (RRs) which receive address translations in RR payloads are automatically set to zero.

Cisco IOS NAT does not translate IP addresses embedded in DNS zone transfers.

I have begun planning for an IOS upgrade to our Supervisor 720s in our Cisco 7600.

Here are a few documents on the process including a doc on upgrading a Cisco Content Switch Module or CSM.

CSM Upgrade Doc

Sup 720 Redundancy Doc

Sup 720 Upgrade Doc

I have put a service-policy in place on my companies high speed WAN link to control how much bandwidth the backup servers can use. This is a time-range based policy so that at night the backups can use as much bandwidth as is available and during the day it can only use 40 Mbps.

Scrubed config

time-range backups

periodic daily 8:00 to 18:00

access-list 100 permit ip host #.#.#.# (Backup Server) any time-range backups

class-map match-all bkup

match access-group 100

policy-map bkup

class bkup

police 40000000 937500 937500 conform-action transmit exceed-action drop

class class-default

fair-queue

int vlan #

service-policy output bkup

I believe the reason this was happening was that the route-map I had in place for another node on this VLAN was causing packets to be dropped. What I did as a more permanent fix was to had an entry in the route-map’s match address list for the problem server. This route-map directs the traffic for those two nodes over a different VLAN between the locations.

I am not sure why the route-map was dropping the traffic and even worse why an access-list allowing traffic from that host would fix the problem. One of those weird issues I guess.

Old Post:

http://blog.super-networking.net/?p=83

I ran into an interesting problem today on our 7600 series router. One device on this router is unable to send traffic to anything on a particular subnet. There are lots of devices on the same VLAN that are about to access that same subnet and this device was up to the other day able to as well.

There is no access-list in between the two sites since the troublesome subnet is in another location. There doesn’t appear to be any routing issues on either router. The server that is having the issue appears to have all the right settings. We have found a weird workaround for this problem.

We are able to get the traffic to pass if we apply an access-list with a permit ip host (server ip) any and a permit ip any any in the in direction on the VLAN that the server in on. This causes the traffic to flow, if you remove the access list the traffic for this one server is once again not able to communicate to this one subnet. If I apply an access-list to this VLAN with just the permit ip any any the traffic will not flow either. All other interVLAN traffic to and from this server work just fine all the time. I have never see this before and for now have left the workaround in place.

If anyone else has run into this problem will reply to this post. I will update this will info as I get it.

The performance issues we were running into with our NAS across a 50Mbps WAN link have been fixed. This issue ended up being a flaky Fiber ONU on one end of the WAN link. The ISP replace this ONU with a Cisco 3400 switch and all of our problems went away.

In performance tests the best we could get on this 50Mbps link was 25Mbps at any one time. The connection is burstable to 100Mbps. After the ONU was replaced we are now seeing speeds up to 86Mbps.

Link to old Post on this Issue:

http://blog.super-networking.net/?p=76

Had an issue this morning where our default route was pointed down our failover ISP connection. All traffic is supposed to go over our main ISP connection which is a flat rate connection and then we have a failover connecton that charges us on usage. I came in this morning and a ton of traffic was going over our failover connection and through the show ip bgp found that our default ISP peer was missing from the default route.

I reset the connections with all of our BGP peer using the clear ip bgp #.#.#.# (peer ip) for each one and the BGP routes came back correctly and all the traffic moved back to our default ISP.

We have been running into performance issues in our datacenter lately that doesn’t show many symptoms except that things just don’t seem to be running fast enough. Mostly I have had a feeling a while that something wasn’t right. We have also been having slowness issues with our NAS, in packet captures we have seen a lot of dropped packets and retransmits.

Here is what I have seen from Cisco on the issue:

Interface/Module Connectivity Problems

Connectivity Problem or Packet Loss with WS-X6548-GE-TX and WS-X6148-GE-TX Modules used in a Server Farm

When you use either the WS-X6548-GE-TX or WS-X6148-GE-TX modules, there is a possibility that individual port utilization can lead to connectivity problems or packet loss on the surrounding interfaces. Especially when you use EtherChannel and Remote Switched Port Analyzer (RSPAN) in these line cards, you can potentially see the slow response due to packet loss. These line cards are oversubscription cards that are designed to extend gigabit to the desktop and might not be ideal for server farm connectivity. On these modules there is a single 1-Gigabit Ethernet uplink from the port ASIC that supports eight ports. These cards share a 1 Mb buffer between a group of ports (1-8, 9-16, 17-24, 25-32, 33-40, and 41-48) since each block of eight ports is 8:1 oversubscribed. The aggregate throughput of each block of eight ports cannot exceed 1 Gbps. Table 4 in the Cisco Catalyst 6500 Series 10/100- & 10/100/1000-Mbps Ethernet Interface Modules shows the different types of Ethernet interface modules and the supported buffer size per port.

Oversubscription happens due to multiple ports combined into a single Pinnacle ASIC. The Pinnacle ASIC is a direct memory access (DMA) engine that transfers packets between backplane switching bus and the network ports. If any port in this range receives or transmits traffic at a rate that exceeds its bandwidth or utilizes a large amount of buffers to handle bursts of traffic, the other ports in the same range can potentially experience packet loss. The buffer assignment on these modules is documented in Buffers, Queues & Thresholds on Catalyst 6500 Ethernet Modules.

A SPAN destination is a very common cause since it is not uncommon to copy traffic from an entire VLAN or multiple ports to a single interface. On a card with individual interface buffers, the packets that exceed the bandwidth of the destination port are silently dropped and no other ports are affected. With a shared buffer, this causes connectivity problems for the other ports on this range. In most scenarios, shared buffers do not result in any problems. Even with eight gigabit attached workstations, it is rare that the provided bandwidth is exceeded.

The WS-X6548-GE-TX, WS-X6548V-GE-TX, WS-X6148-GE-TX, and WS-X6148V-GE-TX modules have a limitation with EtherChannel. For EtherChannel, the data from all links in a bundle goes to the port ASIC, even though the data is destined for another link. This data consumes bandwidth in the 1-Gigabit Ethernet link. For these modules, the sum total of all data on an EtherChannel cannot exceed 1 Gigabit.

Check this output in order to verify that the module experiences drops related to over utilized buffers:

• CatOSCat6500 (enable) show asicreg pinnacle errCheck this output in the list of registers. If the settings in this output are non-zero, it indicates that there were drops due to the buffer overrun.015B: PI_PBT_S_QOS3_OUTLOST_REG = 0011

  015F: PI_PBT_S_HOLD_REG = D26C

• NativeIOSCat6500# show counters interface gigabitEthernet | include qos3Outlost51. qos3Outlost = 768504851

Run the show commands several times to check if asicreg steadily increments. The asicreg outputs are cleared every time they are run. If the asicreg outputs remain non-zero then this indicates active drops. Based on the rate of traffic, this data might need to be collected over several minutes in order to get significant increments.

Workaround

Complete these steps:

1. Isolate any ports that might be consistently oversubscribed to their own range of ports in order to minimize the impact of drops to other interfaces.For example, if you have a server connected to port 1 which is oversubscribing the interface, this can lead to slow response if you have several other servers connected to the ports in the range 2-8. In this case, move the oversubscribing server to port 9 in order to free up the buffer in the first block of ports 1-8. On newer software versions, SPAN destinations have the buffering automatically moved to the interface so it does not impact the other ports in its range. Refer to Cisco bug IDs CSCed25278 ( registered customers only) (CatOS) and CSCin70308 ( registered customers only) (NativeIOS) for more information.
2. Disable head of line blocking (HOL) which utilizes the interface buffers instead of the shared buffers.This results in only the single over utilized port having drops. Since the interface buffers (32 k) are significantly smaller than the 1 Mb shared buffer, there can potentially be more packet loss on the individual ports. This is only recommended for extreme cases where slower clients or SPAN ports cannot be moved to the other line cards that offer dedicated interface buffers.
   • NativeIOSRouter(config)# interface gigabitethernet Router(config-if)# hol-blocking disableOnce this is disabled, the drops move to the interface counters and can be seen with the show interface gigabit command. The other ports are no longer affected provided that they are also not individually bursting. Since it is recommended to keep HOL blocking enabled, this information can be used to find the device that overruns the buffers on the range of ports and move it to another card or an isolated range on the card so HOL blocking can be re-enabled.
   • CatOSConsole> (enable) set port hol-blocking disableOnce this is disabled, the drops move to the interface counters and can be seen with the show mac command. The other ports are no longer affected provided that they are not also individually bursting. Since it is recommended to keep HOL blocking enabled, this information can be used to find the device that overruns the buffers on the range of ports and move it to another card or an isolated range on the card so HOL blocking can be re-enabled.

Also here is a document http://super-networking.net/downloads/6500seriesmodules.pdf on what switch modules to use for what on your 6500 and 7600 series Cisco devices.

I know a lot of the time you see the no ip proxy-arp on your Cisco routers and I guess I have never really paid that much attention to it. Well in some troubleshooting on our network lately I thought I would look into the meaning of this proxy-arp command. Here it is

Proxy ARP is the technique in which one host, usually a router, answers ARP requests intended for another machine. By “faking” its identity, the router accepts responsibility for routing packets to the “real” destination. Proxy ARP can help machines on a subnet reach remote subnets without configuring routing or a default gateway. Proxy ARP is defined in RFC 1027

Advantages of Proxy ARP
The main advantage of using proxy ARP is that it can be added to a single router on a network without disturbing the routing tables of the other routers on the network.

Proxy ARP should be used on the network where IP hosts are not configured with default gateway or does not have any routing intelligence.

Disadvantages of Proxy ARP
Hosts have no idea of the physical details of their network and assume it to be a flat network in which they can reach any destination simply by sending an ARP request. But using ARP for everything has disadvantages, some of which are listed below:

• It increases the amount of ARP traffic on your segment.
• Hosts need larger ARP tables to handle IP-to-MAC address mappings.
• Security may be undermined. A machine can claim to be another in order to intercept packets, an act called “spoofing.”
• It does not work for networks that do not use ARP for address resolution.
• It does not generalize to all network topologies (for example, more than one router connecting two physical networks).

Basically turning proxy-arp on and using it is for people who are too lazy or don’t know how to setup a network. If you don’t see no ip proxy-arp in your config it is turned on so turn it off. It is on by default on some routers. When I turned it off on our network I found a single host that wasn’t setup right and was using proxy-arp. It is now fixed.

Udld is a Cisco proprietary protocol that is used to prevent a unidirectional black hole for traffic. This protocol is meant to be used because Cisco switches with multiple paths. This protocol is not meant to be turned on between a Cisco switch and other non-Cisco devices. The reason I looking into this is that we had this enabled on some links between our Cisco gear and out NAS storage. There was no way this protocol could have functions correctly and was just causing overhead on the link.

Here is a whitepaper on the topic to take a look at.