Super-Networking Blog

So if you are multi-homed on BGP and you want to be able to decide yourself which path incoming and outgoing your traffic will take you will need to make a few changes. By default BGP will try to make the best decision on what path to take. Problem is a lot of time it will just choose the lowest AS number which might not be the best route.

router bgp 11111
neighbor 2.2.2.2 weight 500
neighbor 3.3.3.3 weight 400
neighbor 3.3.3.3 route-map prepend out

ip prefix-list aggregate seq 5 permit 1.1.1.0/24

route-map prepend permit 10
match ip address prefix-list aggregate
set as-path prepend 11111 11111 11111 11111 11111

Under your BGP config on your router add some weight statements, the higher the weight the more preferred route for outgoing traffic. The route-map prepend out statement you will want to put on the neighbor that you don’t want traffic coming in on. Incoming BGP traffic will always try to take the smallest AS number first so if you prepend your AS number onto your neighbors AS path it will seem like the worst path. This will give you the effect of incoming traffic on the neighbor without the prepend router-map.

So I just started a job for a new company. One of the first things I did was look for easily resolvable issues on the network. Some symptoms to look for are any packet loss or high latency on a local LAN link, CRC errors on your switch ports and router ports, ultra slow RDP connections, or slow web surfing. These are just some places to start.

One issue I ran across at my new job that was causing slow web access was a duplex mismatch on our border router. One side was set to static 100/full and the other end was set to auto. You may thing that auto will detect the other end is at 100/full but this is not the case. If both ends are not running at auto they cannot negotiate the speed and duplex settings. In turn the auto end will randomly guess what speed and duplex to run at causing packet loss and slow downs.

To take this a step farther I want to share my experience with what to put these settings at. For point to point cables running at less than 1Gbps you should set both sides to static 100/full. For servers that are running 100Mbps NIC cards or 100Mbps switch ports you should set both sides to 100/Full. You could set both ends to auto but when the server is under stress it may negotiate down to 10/full causing major slowdowns. Client workstations and the switch ports they are plugged into should always be auto because you never know when you will plug a new device into it and you would have to set every new device to 100/full to work correctly. If you are running at Gbps speeds you have to leave both sides at auto, according to the Gbps standard you should never set to static 1000/Full. Most network adapters to not even allow you to. There is a setting on newer Gigabit drivers that allow for 1000/Auto which is really auto negotiate but it is weighted to 1Gbps.

Another thing that can cause slow Internet browsing on the network is failed or misconfigured DNS forwarders. So most people on a Microsoft AD domain use the domain controllers for DNS on the clients. Then you setup external DNS servers as forwarders on the domain controllers. What happens is when a client tries to go to www.google.com for example the client will put a DNS request to the domain controller who will not know it and send it on to the DNS forwarder. The DNS forwarder will then relay the correct information through the domain controller back to the client. If the DNS forwarders on the domain controllers are inaccessible, misconfigured, or having large packet loss your web experience will be slow or not work at all.

Another one I ran across lately was a server plugged into the local LAN that we returning a latency of 3-4ms per ping. This should never be above 1ms for any period of time unless that server is under an extreme load and even then it is rare. More likely there is another problem. I checked speed/duplex settings and they were fine, replaced the cat 5 cable, changed switchports none of which worked. I logged into the console and it was extremely slow, we are talking 10mins to login and even one you are logged in it took forever to do anything. I rebooting into dos mode with IP connectivity and the pings were still 3-4ms. It was an older box and I figured it was a hardware issue, I had a spare box of the same specs, moved the drives and when it came back up it was 10 times faster. Also the pings we less than 1ms.

Another one I found this past week was a server that was having problems talking to the domain. You could RDP to it but could not login because it couldn’t talk to the domain. I ran a ping against it and I noticed about every 3rd packet was dropping. I replaced the cable and everything worked again.

If you have run across any network troubleshooting situations lately comment on the blog entry for others to learn.

So for the past couple of days I have been playing cable monkey at work. I found a hub in our server room that was running 12 servers with one uplink to a switch. Keep in mind this is a hub. Hubs should never be put on a business network now a days especially with servers plugged into them. So I got a Cisco 2960 48-port switch and ran 16 runs of new cable to replace the hub.

Everything was going pretty well, got the cable ran and the switch installed and running. I then started to terminate all of the cables which is never fun but typically not an issue. Today it was an issue, 75% of the cables I terminated were not testing out. I replaced the ends and still would not test out. It was typically the same pairs of wires that were having the issues too.

My first though besides me going crazy and forgetting how to cable was that the spool of cable I had used for the runs had bad cable in it. I also thought maybe the cable tester went bad. I was about to rule that out by testing some patch cables. After many attempts at trial and error I had a Sysadmin, who knew how to cable, check it out. He noticed that on the bad cables some of the metal prongs were not going down. He tries to reclamp down on the ends with no luck either. It appears that the crimpers I was using had gone bad, I wasted a lot of Cat 5 ends. Tomorrow I will get a new crimpers and hopefully be able to finish the job.

I did have a personal pair in my laptop bag but they are currently missing.

So I have started a new job for a new company in the past week. I will not name companies because I don’t believe they have to be mentioned. People who I have worked with in the past know my past companies I have worked for.

I have worked under four IT managers in the past 8 years and 2 of them were good. Thankfully, one of those four managers I am currently working for. The first IT manager I worked for was a terrible manager, knew very little about IT topics and refused to listen to his staff. The company I worked for at the time had a CTO that saw something had to be done with this manager for the good of the company and the good of the IT dept. This IT manager was moved to another department and I got a new IT manager. He was a great manager in my opinion, he listened to his IT staff, he empowered his IT staff, and he backed his IT staff. I learned a lot from him, how to manage people, how to handle day to day politics, how to get things done.

I loved working at this company for this manager, I unfortunately had to leave this company because the they were bought out. I moved to another company in the area that sounded great. With their talks of bonuses, trips, and free pop. In the interviews the IT manager talked about how cutting edge they were and all of the huge projects they were working on. I joined their team, moved into the lead Network Administrator position. From the first week that I was working for this new company I knew there were issues in the department.

My first week another Network Administrator quit without notice basically saying he could not work another day under this particular IT manager. In the following weeks I saw blow-up after blow-up of different IT employees when they had run-ins with their manager. Things rolled along somewhat ok for the first 6 months I was there because we were really busy. The company was building a new corporate building and we need to set it all up. Once we moved into the new building and really settled in the issues that had been brewing really started to explode. More and more highly stressed blowouts were occurring among staff. It got to the point where a number of staff went to the COO about it.

The environment at this company in the IT Dept continued to degrade, projects of all kinds broke down. Nothing new got approved without overly drawn out battles with the IT manager. Projects pushed by the manager just wasted everyone’s time. Larger project would get restarted halfway through by the manager and shifted from person to person throughout the staff. The entire department basically lost the will to work, a 2nd person in upper management  was involved with the issues and still nothing change. IT personnel started to jump ship, 9 of the 11 staff within 9 months. A 3rd person in upper management was involved and still nothing changed.

It is my opinion that a bad manager is a bad manager, it doesn’t matter what training you send them to or how many discussions you have with them. You can micro-manage them and still they will be a poor manager. Their staff will know this and once the staff loses all respect for their manager the game is over the department has broken down. You can do one of two things as upper management, you can ignore the reality and try to fix the manager, you can hope it gets better, even throw money at it. The other thing you can do it make the hard move, replace the manager, hopefully before too much damage is done.

Thankfully the first company I worked for that had a terrible boss realized what needed to be done and was able to turn around the dept almost overnight. Instead the second company decided to look the other way as their department was decimated by the in battles between staff and management. Is the feelings of one person that much greater than the lives of 9 others and the stability of the company as a whole?

To my fellow past co-workers: Brad, Brye, John, Mike, Mike, Allen, Chee, Calvin, and Josh good luck in the future. I am sure your future jobs will be brighter than you past. Garry I hope things can turn around there before things are too late for you. 

As for myself I look to the future and it shows great potential at my new company. I have seen in only my first 5 days that things are back to the way they should be. I am getting things done, getting projects moved forward and making a difference. My boss listens to my opinions and has faith in his staff.