Super-Networking Blog

One problem that anyone who runs a server room or a datacenter has come across is how do I keep it cool. Servers put off a ton of heat, now with the popularity of 1U servers and Blade Servers the heat problem has only gotten worse.

The best way to keep a datacenter cool is to design it correctly in the beginning. Get large enough A/C units, calculate the amount of heat being put off by your equipment. Have professional cooling technicians come in and design the perfect solution. That is fine if you are designing a brand new datacenter but what do you do if you have an old datacenter that has grown beyond its cooling capability.

First off you should do what you can to direct the hot and cool air in your server room/datacenter. You should try to setup cool rows and hot rows, you don’t want to be kicking hot air from one rack into the intakes of servers in another rack. Always exhaust hot air away from all other equipment.

Next do not cool the hot air directly. It is very common to put a cold air outlet vent right over the exhaust side of your server racks. You are trying to cool the hot air before it gets to the rest of the room. The problem with this is it isn’t very efficient. The better way to cool your datacenter is to vent the cold air into the space the server intakes are drawing from. You then want to put your A/C unit’s hot air intakes right above your rack’s exhaust side. What you are doing here is sucking as much of the hot air as you can out of the room as soon as it comes out of the rack, you are then putting the cool air in place for the servers to take it in. This works well because you are cooling the ambient temperature of the room and getting rid of the hot air instead of wasting all your energy cooling really hot air. Also with the servers pulling in colder air the servers should stay cooler even if the hot rows warm up a lot.

Try it out, it works quite well when you are on a tight budget.

We have started to look into purchasing long haul Dark Fiber in an ever developing effect to get lower cost high bandwidth.

Definition of Dark Fiber (by Wikipedia)

“In fibre-optic communications, dark fibre or unlit fibre (or fiber) is the name given to individual fibers that have yet to be used within cables that have been already laid. They are hence not yet connected to any device, and are only there for future usage.

The term was originally used when talking about the potential network capacity of telecommunication infrastructure, but now also refers to increasingly common practice of leasing fiber optic cables from a network service provider.”

I have been putting out feelers to different companies about leasing some dark fiber and I will update when I find out more.

Good backround info here

Well we had a jam-packed day full of datacenter tours yesterday. We drove all over Minneapolis and St. Paul to look at 6 different datacenters. 10 Hours of touring and talking about places to co-locate some of our equipment.

All of the datacenters we visited were quite different, both in the facilities and the people of managed them. I am going to go through some of the highlights of all the datacenters we visited yesterday.

-TWTC (511 Building, Minneapolis)

We took a look at their old co-location space and the new 10,000 sq foot space they were building in the 511 building. They had a pretty nice layouts, the air conditioning wasn’t bad, the security wasn’t bad.

They don’t offer any managed services but it you need a cheap place to locate some equipment and you have the man power to manage it go for it.

Still nicer than having your own datacenter since they take care of the AC/Power/Security of your equipment for you. They have generator backup power, UPSes, no raised floor, fire protection was dry pipe only.

-Sunguard (511 Building, Minneapolis)

We took a look at their enterprise class datacenter in the 511 building and it was pretty impressive. They take a lot of steps to ensure security of your equipment, they have an onsite 24/7 manned NOC onsite.

Everything was very clean, very professional. They pushed their professional services a lot and it was clear that they wanted you to use all of their managed services you could because that is where they make their money.

They had plenty of air conditioning with a raised floor, they had a fire suppression system as well as dry pipes. Generator backup power, redundant everything.

Very nice datacenter.

-Honeycomb (Minneapolis)

We took a look at this datacenter and I would have to say it was the most interesting setup we saw. You pull into a neighborhood that doesn’t look like a co-location space should live there.

The building is a 1900 century in not good condition place that you walk up very narrow dangerous stair to get to. Basically what it is, is a geek shop that someone gutted some apartments, put up lots of AC, power, racks, generate etc… to start up a co-location space.

This is a total non corporate geek place where you can get space and managed service for much cheaper than you are going to get at a Sunguard. They are very rough around the edges, but seemed to be quite knowledgeable, they are staff 24/7 and had some good bandwidth coming in.

Did have a fire suppression system. Seemed a little warm in their datacenter for my taste.

Visi (St. Paul)

We took a look at this enterprise class datacenter and I was really impressed. They seemed to have it all, they has more than enough air conditioning, they had a ton of space available and even more to expand later.

They had raised floors for their AC, they had plenty of power and redundant everything, Fire suppression, etc…. They provided everything from just a 1/4 of a rack to caged off areas for all your racks. They will give you any level of managed service you need up to a completely hosted solution.

Great security, clean place, friendly people. They have a 24/7 NOC on site that is always managing the datacenter. Everything was straight forward, didn’t push anything and we will to work with you.

I would rank this datacenter #1 out of all the datacenters we toured yesterday.

Implex (Minneapolis)

This datacenter is in the basement of a high-rise building in Minneapolis. They datacenter is located in a bank vault for extra security. This datacenter was rather disappointing to me, their datacenter was downright hot, most equipment was in open racks with cables running everywhere.

Didn’t seem like they had their act together at all, someone could trip and rip a cable out of your equipment or power something off accidently. They didn’t have a lot of room open but even if they did their AC wouldn’t have been able to handle it. Wasn’t really happy with any aspect of their co-locate space.

I would rank this datacenter as the worst one we toured yesterday.

ipHouse (Tritech Building, Minneapolis)

This datacenter is located in the Tritech building and was impressively large. This was by far the largest open facility that we toured but most of the space was open cages for growth.

As I said this was a large space and so hard to cool and they did appear to have some cooling issues. They didn’t have a raised floor and really just seemed too hot for me to be real comfortable.

They have good security, redundant power, generators , etc… There are a ton of ISPs/telecoms in this area for cross connecting your connection to different carriers would be no problem. The people running this datacenter were knowledgeable, friendly, and willing to work with different situations.

I think I would rank this datacenter as my #2 pick over the ones we saw yesterday.

I would imagine that they are going to be much cheaper than Visi my #1 pick so might be the best viable solution.

We haven’t gotten prices back from any of these places and that could change some of the standings but I think by taking a look at these datacenters we got a feel for how they are going to stack up price wise.

This by far isn’t all the details for these datacenter but the things that stuck in my mind. If you have any questions of comments about these datacenters post a comment.

I guess there is a new concept being shown at cebit for protecting your datacenter from fire. You reduce the amount of oxygen that is in the air in your datacenter. Air typically has about 21% oxygen in the air and you can keep most fires from starting by reducing the oxygen levels to around 16%.

This is supposed to still be safe for human to breath and it like being at a very high altitude. Great now we get to put up with bone chilling temps and low oxygen. Sounds like a cool idea, will have to wait for more details on this.

Another article on this. This article is a little more questionable, they seem to be trying to say that by reducing the oxygen level it prevents overheating. I doubt it.

Listen to your Voicemail Online

I read an article today in the St. Cloud Time that was about the benefits of running you tech company out of St. Cloud Minnesota and almost fell off my chair laughing.

The only way to benefit from putting your datacenter in downtown St. Cloud is if you are hosting it somewhere else in Central Minnesota. You are much better off putting it in a co-location space in Minneapolis/St. Paul or even in Chicago. The bandwidth costs to get high bandwidth solutions into St. Cloud are very expensive, the power solutions in downtown St. Cloud are not good and most of the buildings are in disrepair.

My suggestion is if at all possible consider renting space out of the Twin Cities. If you need to stay in the St. Cloud area and do need higher bandwidth then by all means stick as close to the Qwest building downtown as you can.

Listen to your Voicemail Online

We have started to look for a new server co-location center in the Minneapolis/St. Paul area. There are quite of few of them to take a look at. We are probably going to start taking tours soon so if anyone has recommendations on co-location spaces in the Twin Cities area please comment here.

We are looking for locations that can provide us high bandwidth at good costs, good hosting provider customer/technical support since we are not local. Of course there are a lot of factors but these are a couple of the higher ones on the list.
Some places we are going to look at:

Vericenter

Implex

ipHouse

XO Colocation

VeriSpace

Honeycomb

Visi

Sunguard

TWTC

Check out my new Cabling Info Page. It is always hard to know if on your next server install should I run Cat 5e cable or Cat 6? Should I buy Cat 6 patch cables? Is Cat6 worth the extra price compared to Cat 5e? Will Cat 5e run GigE, what about 10GigE? Well not all of the questions are answer in the white papers I put on this site but a good portion of them are. There is some good information on the lifetime cost of running Cat 5e vs. Cat 6 vs. Cat 7.

One thing I have noticed just recently is companies like Best Buy are not even selling Cat 5e anymore, Cat 6 is the future and its prices are coming down fast. So when you get a chance check out this new page. One other comment I would like to make because I have seen this done before. If you can’t terminate cable to spec, don’t cable and don’t buy Cat 6. No matter how much shielding your cable has if you don’t put the ends on right your cable is worthless.

I have a question to pose to everyone and I will give my take. What is redundancy? If you use the term redundant and datacenter in the same sentence how true is the statement. If you have two web servers that are load balanced but both of them point back to one database is that redundant? If you have multiple power supply each plugged into its own UPS but no generator is that redundant? If the core systems like cooling and power aren’t redundant how hard should you work at making the systems redundant? How hard should you work on the front end to make it redundant if the backend is a single point of failure? My option is that none of those situations are redundant, maybe load sharing but not redundant. If something as simple as A/C and power is not redundant in a datacenter I wouldn’t worry about spending thousands of dollars and add a ton of complexity for something that will break anyway. I would like comments on this, am I way off base? The question I pose to people with a “redundant” system without backup power or A/C is what is more likely? A Cisco switch/router burning out or the power going out? What about the A/C unit freezing up and causing the datacenter to overload, that sure happens a lot.

I got another comment on my earlier blog Click Here on different network gear working well together. Most of my experience is with straight Cisco gear but in dealings with various equipment from various vendors I would say just stick with a standard. If you want to run all 3COM switches or all Intel switches just stick with it. I would stay away from the Dells because they don’t even seem to follow standard between their models. Also as far as a business I would never got with a consumer grade product like Linksys or Dlink. Cisco does own Linksys but it is “consumer grade” and will not hold up well in a business environment. Also if you are just doing a flat network with no VLAN tagging, no trucking nothing fancy it makes less of a difference, if you are going to be putting multiple VLANS on a switch, setting up etherchannels, etc… go with a quality product like a Cisco.