Super-Networking Blog

So I have been doing BGP on Cisco routers for about 6 years or so, it is a really good way of having a redundant ISP connection. Sure it can be a pain to get it setup as some ISPs take a while to get the settings right but once it is setup and tested normally it works like a charm. Typically I run the 2 ISP connections in an active/standby fashion instead of load sharing. This allows you to get a better rate on the backup connection as long as you don’t use them on a regular basis. Not using both connections is easier said then done though, when you setup BGP by default it will choose the best route for the connection so it will use both connections. In the past I have done my best to fix this problem by weighting the connections both in and out so I can choose the best path for outgoing connections and advertise the best path for incoming connections. Doing it in this manner works pretty well but their is always some traffic on the incoming side of the redudnant ISP for destinations that are directly connected to that ISP.

I have never had a problem with this lower amount of traffic before from that 2nd ISP but I was tasked recently with finding a way to bring that traffic down to almost zero. What I found was BGP Conditional Advertisement Feature that Cisco has. I had never heard of this before but sounds like the perfect solution to my problem. Basically what this feature does is allow you to setup a conditional BGP advertisement to the 2nd ISP connection. In this condition you put a route or some other dynamic information you recieve from ISP 1 and your normal advertisments for ISP 2. If this condition sees the information you specified from ISP 1 it will withhold the advertisements from ISP 2. If the information from ISP 1 goes away the condition will then advertise to ISP 2. I have not put this in place yet as I am still looking into aspects of putting it in place. This might cause a longer outage when ISP 1 fails but will solve the problem of incoming traffic on ISP 2 because if I am not advertising at all out ISP 2 I can’t get any traffic from them.

Here is a document on how to configure this feature on you Cisco Router.

http://www.cisco.com/en/US/tech/tk365/technologies_configuration_example09186a0080094309.shtml

Do you want your workstations to connect to PPTP VPNs through a Cisco Pix firewall without having to setup a static NAT for each one.

Are you getting the following error in your syslogs when you try:

“regular translation creation failed for protocol 47″

All you should have to do is add a new fixup protocol entry.

“fixup protocol pptp 1723″

Now assuming that you have a PAT for all traffic from inside to outside you PPTP connections should work.

There are other factors that could cause this to fail, make sure that you are on at least version 6.3 of the Pix software.

If you still have problems check your syslogs.

Holy cow! Can you believe that finally the developers of Wireshark have reached 1.0.0 of their product. I have been running 0.9X versions of this software for what seems like forever.

Wireshark for you who haven’t looked for a long time used to be the packet sniffer called Ethereal. This is a great free packet sniffer that every Network Admin worth his salt should have on his laptop and or Thumb Drive.

Download it here

Older Post on Wireshark/Ethereal:

Wireshark aka Ethereal

Do you need something to monitor the bandwidth usage on one server? For instance if you have a hosted server that has limited total usage. Well if you want a simple free tool download NetMeter.

You can get real time statistics, totals including day, week and month. It will also project your usage over the day, week or month based off what you have already used. It is easy to use and best of all FREE!

If you really want to configure your IP address from the command prompt instead of the Windows GUI you can do so with the Netsh.exe command.

To see your current network config go to a command prompt and type in the following command “netsh interface ip show config”

If you want to set you IP to 10.10.0.2 with a subnet mask of 255.255.255.0 and a gateway of 10.10.0.1 on your Local Area Connection adapter use the following command: “netsh interface ip set address name=”Local Area Connection” static 10.10.0.2 255.255.255.0 10.10.0.1 1″

Want to setup DNS settings: “netsh interface ip set dns “Local Area Connection” static 10.10.0.200″

These commands should work on Windows Server 2003 and Windows XP, might work on other versions too but haven’t tested.

So I thought I would discuss the topic of multiple default gateways on Windows Servers today since my buddy blogged about it.

IT and Development Best Practice

First off you should never have two default gateways on your servers. It doesn’t matter which OS you are running, Windows 98/NT is worse at handling the condition but all of them will have problems. Every windows server has a little routing table in memory. If you have one network card it will know to send anything on its own subnet out that adapter even if you don’t have a gateway. It doesn’t know how to get to anything outside of its own subnet without a default gateway. Now if you have two adapters without default gateways the server will know to send traffic for each subnet depending on which it tied to which network card.

When you add a default gateway to a network card the server will then send anything outside of its local subnet(s) to that gateway you specify. The gateway, being a router or firewall, which then pass the traffic on to the correct location. The problem comes in if you have default gateways on more than one network card. The server then gets confused on where to send its traffic when it is off its local subnets.

The way it is supposed to work is the adapter that is bound first to the OS should be the one used but that isn’t what happens. Instead the OS randomly guesses at which gateway to use and some packets may get through some will not. This is not only for different sessions but in mid session as well.

So what do you do when you have a dual-homed server, you want multiple subnets to go out one adapter and everything else to go out the other? You add the default gateway on the adapter that you want the traffic to go when you might not know the subnet like say Internet traffic. Then add a route statement for the subnets you know and want to go out the other adapter.

Example:

Adapter #1 - 10.1.1.20 Mask 255.255.255.0 Default Gateway 10.1.1.1

Adapter #2 - 192.168.2.20 Mask 255.255.255.0

Go to a command prompt and add the below command

“route add 192.168.0.0 mask 255.255.0.0 192.168.2.1″

This would tell the server that all traffic that is heading for anything on the 192.168.0.0 subnets would be directed to the IP 192.168.2.1. That adapter would not have a default gateway. The other adapter has the default gateway and all other traffic would go out that adapter and to the IP specified. Note that the command above will go away on restart unless you specify -p and it will stay until you do a route delete.

If you want to see all the routes your server is currently using run the command “route print”

One thing to ask yourself before going through this, do I really need a dual-homed server? What security holes am I opening up? Am I bypassing a firewall?

So as many of you know who have used it in the past netflow is a great tool. Netflow gives you detailed information about traffic flowing through your routers. You can find out what IPs the traffic is coming from and going to, you can see what protocols, what ports and how much traffic is going through that router. Big help if you are trying to find what a traffic spike is or why your Internet connection is maxed out.

Typically you need software to collect the exported netflow data and then compile it into some easy to read form. This software isn’t typically cheap or what if you need it now and don’t have time to install a collector. I will give you the commands you need to get a quick look at the traffic flowing right now.

First thing you have to do is have the router watch the flows:

Under each interface type the command “ip route-cache flow”

Exit the interface config and add the command “ip flow-export version 5” to select a version.

Let is collect traffic for a little bit, shouldn’t impact the performance or your router.

Then to see the traffic break down type the command “show ip cache flow”

This will give you the basic traffic breakdown going through you router. Look at Pkts column to see where the heavy hitters are. When you are down looking at it just leave it running on your interfaces, won’t affect performance and will be in place for when you have a netflow collector installed.

So if you are multi-homed on BGP and you want to be able to decide yourself which path incoming and outgoing your traffic will take you will need to make a few changes. By default BGP will try to make the best decision on what path to take. Problem is a lot of time it will just choose the lowest AS number which might not be the best route.

router bgp 11111
neighbor 2.2.2.2 weight 500
neighbor 3.3.3.3 weight 400
neighbor 3.3.3.3 route-map prepend out

ip prefix-list aggregate seq 5 permit 1.1.1.0/24

route-map prepend permit 10
match ip address prefix-list aggregate
set as-path prepend 11111 11111 11111 11111 11111

Under your BGP config on your router add some weight statements, the higher the weight the more preferred route for outgoing traffic. The route-map prepend out statement you will want to put on the neighbor that you don’t want traffic coming in on. Incoming BGP traffic will always try to take the smallest AS number first so if you prepend your AS number onto your neighbors AS path it will seem like the worst path. This will give you the effect of incoming traffic on the neighbor without the prepend router-map.

So I just started a job for a new company. One of the first things I did was look for easily resolvable issues on the network. Some symptoms to look for are any packet loss or high latency on a local LAN link, CRC errors on your switch ports and router ports, ultra slow RDP connections, or slow web surfing. These are just some places to start.

One issue I ran across at my new job that was causing slow web access was a duplex mismatch on our border router. One side was set to static 100/full and the other end was set to auto. You may thing that auto will detect the other end is at 100/full but this is not the case. If both ends are not running at auto they cannot negotiate the speed and duplex settings. In turn the auto end will randomly guess what speed and duplex to run at causing packet loss and slow downs.

To take this a step farther I want to share my experience with what to put these settings at. For point to point cables running at less than 1Gbps you should set both sides to static 100/full. For servers that are running 100Mbps NIC cards or 100Mbps switch ports you should set both sides to 100/Full. You could set both ends to auto but when the server is under stress it may negotiate down to 10/full causing major slowdowns. Client workstations and the switch ports they are plugged into should always be auto because you never know when you will plug a new device into it and you would have to set every new device to 100/full to work correctly. If you are running at Gbps speeds you have to leave both sides at auto, according to the Gbps standard you should never set to static 1000/Full. Most network adapters to not even allow you to. There is a setting on newer Gigabit drivers that allow for 1000/Auto which is really auto negotiate but it is weighted to 1Gbps.

Another thing that can cause slow Internet browsing on the network is failed or misconfigured DNS forwarders. So most people on a Microsoft AD domain use the domain controllers for DNS on the clients. Then you setup external DNS servers as forwarders on the domain controllers. What happens is when a client tries to go to www.google.com for example the client will put a DNS request to the domain controller who will not know it and send it on to the DNS forwarder. The DNS forwarder will then relay the correct information through the domain controller back to the client. If the DNS forwarders on the domain controllers are inaccessible, misconfigured, or having large packet loss your web experience will be slow or not work at all.

Another one I ran across lately was a server plugged into the local LAN that we returning a latency of 3-4ms per ping. This should never be above 1ms for any period of time unless that server is under an extreme load and even then it is rare. More likely there is another problem. I checked speed/duplex settings and they were fine, replaced the cat 5 cable, changed switchports none of which worked. I logged into the console and it was extremely slow, we are talking 10mins to login and even one you are logged in it took forever to do anything. I rebooting into dos mode with IP connectivity and the pings were still 3-4ms. It was an older box and I figured it was a hardware issue, I had a spare box of the same specs, moved the drives and when it came back up it was 10 times faster. Also the pings we less than 1ms.

Another one I found this past week was a server that was having problems talking to the domain. You could RDP to it but could not login because it couldn’t talk to the domain. I ran a ping against it and I noticed about every 3rd packet was dropping. I replaced the cable and everything worked again.

If you have run across any network troubleshooting situations lately comment on the blog entry for others to learn.

So continuing on the config things through the registry topic, not because I am trying to be a hardcore geek but because I need to bootleg something.

So what do you do when you only can get on a box (Server 2003 or Windows XP SP2) remotely and the firewall is enabled. Either out of shear luck or in my case shear skill, you know it, you can stop the Windows Firewall/Internet Connection Sharing (ICS) service. Then you can remote desktop into the box.

Say now that you want to enable the firewall on one of the adapters and not the other. You go into properties on the adapter - > go into Advanced - > Settings. It tells you the service is stopped and if you want to configure it you will need to start the service. If you have it start the service you will be kicked off the box with the firewall now on, back to square one.

So what you do is go into the good old registry:

HKEY_Local_Machine\CurrentControlSet\Services\SharedAccess\Parameters\FirewallPolicy\StandardProfile
-or-
HKEY_Local_Machine\CurrentControlSet\Services\SharedAccess\Parameters\FirewallPolicy\DomainProfile

Change the setting of firewall from on to off. 0-Being Off and 1-Being On. Now start the ICS service and you should stay connected and the firewall should stay off on the box.

Now you can go into properties on the adapter you want the firewall on - > go to Advanced - > Settings - > Click “On” in the general tab then go to the advanced tab and uncheck the adapter you want the firewall turned off on. Ok out of everything and now you have the firewall enable on the adapters you want and off on the others instead of by default all adapters go on/off together.

Best part is you didn’t have to drive to the remote location to do it.