Optimizing a Fire and Forget page tracker

I have a bit of code that came from Egg Head Cafe a long while ago and it's been working great. But I ran the entire site through Redgate's Profiler. and it came up as the biggest hot spot in my code. (Honestly it ran so hot the rest of my site hardly spiked.) The point of it is to capture traffic into my site and filter out crawler traffic if it's needing to be blocked. It's worked greatly for years and I'm wanting to reuse it, but hoping it can be optimized a bit.

Starts out in the Global.asax:

protected void Application_PreRequestHandlerExecute(object sender, EventArgs e)
{
    Logger.LogRequest(sender as HttpApplication);
}

This moves into the Logger class. Not the real name, just changed for the post.

```
public class Logger
{
private static string _conn = ConfigurationManager.AppSettings["MyDatabase"];
private static bool _denyBots = false;

private static void DenyAccess(HttpApplication app)
{
app.Response.StatusCode = 0x191;
app.Response.StatusDescription = "Access Denied";
app.Response.Write("401 Access Denied");
app.CompleteRequest();
}

public static bool IsCrawler(HttpRequest request)
{
bool isCrawler = request.Browser.Crawler;
if (!isCrawler)
{
Regex regEx = new Regex("Slurp|slurp|ask|Ask|Teoma|teoma");//shortened
isCrawler = regEx.Match(request.UserAgent).Success;
}
return isCrawler;
}

public static void LogRequest(HttpApplication app)
{
HttpRequest request = app.Request;

bool isCrawler = IsCrawler(request);
string userAgent = request.UserAgent;
string requestPath = request.Url.AbsoluteUri;
string referer = (request.UrlReferrer != null) ? request.UrlReferrer.AbsoluteUri : "";
string userIp = request.UserHostAddress;
string isCrawlerStr = isCrawler.ToString();

object[] parms = new object[] { userIp, userAgent, requestPath, referer, isCrawler

The BeginInvoke() call is a hotspot because it will need a thread from the thread pool (and when there are too many used, it will wait for one to be available).

As a micro-optimization, try replacing that with ThreadPool.QueueUserWorkItem and measuring again, but in the end you're starting one (or more) sql insert query per request, that will use some resources no matter what is the async approach you use.

You could also use SqlCommand's BeginExecuteNonQuery / EndExecuteNonQuery to do your inserts async (instead of BeginInvoke / EndInvoke or the ThreadPool), but in the end the fastest solution involves not doing the insert to db per request, just write the needed stuff to some kind of queue (preferably one faster than the db) and insert to the db at a later time.

StackExchange Code Review Q#4659, answer score: 5