Super-Networking Blog

What do you do if you want to look for combinations of word strings in Gigabytes of IIS log files but don’t want to spend any money?

You download a program called Strings.

This is a SysInternals program now owned by Microsoft that will search through text files for word strings and kick them out to you.

So I will go through an example of how you can use it and you can modify it from there.

So copy all of the IIS log files to a location, for example c:\logs

Then open up a command prompt, go to the directory where you have the strings program.

Type the command: strings.exe -s c:\logs | findstr “Your Search Text”

This will dump all the lines it finds with that search text to your command prompt window. Now take it further.

Type the command: strings.exe -s c:\logs | findstr “Your Search Text” > c:\output\log.txt

That will dump the found lines to a new text file. Either you can use that text file for whatever you want or parse it down further with the following.

Type the command: strings.exe -s c:\output | findstr “Your Search Text 2″

That will allow you to search through huge directories of IIS logs for combinations of things. If you do the same search often try dumping it into a batch file.

Thanks Goes to Steve For the Idea to Look Into Strings

Do you want your workstations to connect to PPTP VPNs through a Cisco Pix firewall without having to setup a static NAT for each one.

Are you getting the following error in your syslogs when you try:

“regular translation creation failed for protocol 47″

All you should have to do is add a new fixup protocol entry.

“fixup protocol pptp 1723″

Now assuming that you have a PAT for all traffic from inside to outside you PPTP connections should work.

There are other factors that could cause this to fail, make sure that you are on at least version 6.3 of the Pix software.

If you still have problems check your syslogs.

Ran across something interesting this week. I was looking into performance on some of our webservers and noticed that there would be unexplained CPU spikes every once in a while. No real spikes in traffic, no additional tasks going on.

In the event logs I did find something that correlated with the CPU spikes:

Event ID: 7036

“The WinHTTP Web Proxy Auto-Discovery Service service entered the running state.”

Event ID: 7036

“The WinHTTP Web Proxy Auto-Discovery Service service entered the stopped state.”

This happens many times a day, starting and stopping. When in transition it seemed to spike the CPU for a little while.

Service Description is the following:

Implements the Web Proxy Auto-Discovery (WPAD) protocol for Windows HTTP Services (WinHTTP). WPAD is a protocol to enable an HTTP client to automatically discover a proxy configuration. If this service is stopped or disabled, the WPAD protocol will be executed within the HTTP client’s process instead of an external service process; there would be no loss of functionality as a result.

In looking into this more I found this is not needed to run IIS or anything else on your server. Only needed for your browser on your server to detect a proxy for Internet browsing. So do yourself a favor and disable this service, you will see it most likely set as manual and stopped but don’t let this fool you. Check out your event logs and if you see it starting and stopping disable it.

So if you are running Windows 2003 with IIS 6 and all of your web applications are running ASP .Net 2.0 you might want to get rid of .Net 1.1.

Well with Windows 2003 it isn’t easy or advisable to completely remove .Net 1.1 but you can unregister is from IIS having the same affect as far as your websites are concerned.

You need to open up the command prompt and go to the path .Net is installed in. Typically is C:\Windows\Microsoft.NET\Framework\v1.1.4322\

If you have other versions of .net installed you will see all the versions under Framework as well, be sure to go into the 1.1 directory.

From the command prompt in that directory type the command “aspnet_regiis.exe -u”

This will unregister this version of the framework with IIS and in turn IIS will not see it. You will want to do a reboot to make sure everything comes up clean afterwards.

Also make sure that you do not have any websites using .Net 1.1 because you will most likely mess it up when it is forced over to .Net 2.0

So I posted earlier on splitting up the robots.txt you are showing based on if the bot was connecting on http or https here. I commented at the end that if this is an existing site and Google has already indexed your https content you need to be careful. Once you block Google from reindexing your https content using the robots.txt file Google will ignore changes, but it will keep the old indexed pages.

You cannot use the URL removal tool because you can’t tell it to only remove https urls. The way you get around this is to remove the robots.txt restrictions from https and put meta tags for Google to remove the content only on our https content.

Tags you are going to want to user are:

<meta name=”ROBOTS” CONTENT=”NONE”>

and

<meta name=”GOOGLEBOT” content=”NOARCHIVE”>

This will tell Google to not index the pages and remove any old caches and indexes. This could take some time for Google to crawl all of your https content. Once everything you want out of their index is gone you can go ahead and change over to the easier https robots.txt file restrictions.

I do not have the code that allows you to set different meta tag info based on http or https. If someone has that please comment on this post. This should help out the SEO on your site. Thanks

Resource Page for You

One of the big things a lot of people are doing these days is SEO or search engine optimization. I am not planning on going over a comprehensive guide for SEO right now but as I run into them in my own job I will post them here.

One of the more recent tasks I had was setting up a different robots.txt for our http site vs our https site. It is the same content but Google indexes them as two different copies of the content. Google then penalizes your site for duplicate content. Well you setup your robots.txt file to exclude the files and directories you want from the bots but it doesn’t help the http vs http. Both protocols will use the same robots.txt file and so you will end up with duplicate content in Google’s index.

Here is how to solve this problem when you are running IIS 6.0 on a Windows 2003 server. You will need to also be running ASP .Net 2.0 for this solution to work. You might be able to get this to work on other platforms but I have not tested them. What we will be doing here is creating a dynamic robots.txt file, it is only one file but it will display different results depending on if you connect with http or https.

1) Create your robots.txt file:

<%@ WebHandler Language="C#" Class="MyNamespace.robotshandler" %>

using System;
using System.Web;

namespace MyNamespace {

    public class robotshandler: IHttpHandler {

        public void ProcessRequest (HttpContext context) {

            context.Response.ContentType = "text/plain";
            context.Response.Write("User-agent: *\n");

            if (context.Request.ServerVariables["Https"]=="off"){
                // HTTP
                context.Response.Write("Allow: /\n");
                context.Response.Write("Disallow: /MyDisallowedDirectory/\n");
            } else {
                // HTTPS
                context.Response.Write("Disallow: /");
            }
        }

        public bool IsReusable {
            get {return false;}
        }
    }
}

2) IIS needs to have .txt files passed through ASP .Net

- Open IIS and right click on your website and bring up the properties screen

- Go to Home Directory > Configuration. You will be on the Mappings Tab.

- Locate the ASPX item and click Edit - Copy the path in the Executable Field and cancel out of that window.

-Cancel

-Click “Add”

-Populate the Executable path with the value you copied in the last section

-Extension “.txt”

-Enter “GET” in the “Limit To” field

-Save and Exit

3) Need to modify web.config to process the .txt correctly

-Add the following under system.web:  (Look for these sections in your web.config, most likely don’t have to add the httphandler tags because they will be in there already then just add the lines inside them.)

<httpHandlers>

<add path=”/robots.txt” verb=”GET” type=”System.Web.UI.SimpleHandlerFactory” />

<add path=”*.txt” verb=”GET” type=”System.Web.StaticFileHandler” />

</httpHandlers>

<buildProviders>

<add extension=”.txt” type=”System.Web.Compilation.WebHandlerBuildProvider” />

</buildProviders>

Make sure your new robots.txt file is in your root folder for your website and everything should be good to go. Try your website out with http then https and it should be different. You will have to modify the robots.txt file to exclude what folders and files you want.