A new approach to solve SharePoint’s painfully slow spin-up / start-up time for first request

This article was originally posted on our CTO’s personal blog. Due to the tremendous amount of positive feedback we have decided to update the post and move it to the Muhimbi blog, which has more of a SharePoint focused audience. Please check out our PDF Converter for SharePoint, URL Shortener and other SharePoint products if your find this article helpful._

Ever since SharePoint 2007 was introduced I have been really disappointed with the 2 minute spin-up time of the sites and the 30 second wait when launching STSADM.

How could Microsoft have released this? I thought, What is wrong with these people? Why is there no uproar in the SharePoint Community at large?

The SharePoint community quickly came up with workarounds such as the SPWakeUp tool on Codeplex, but these tools don’t actually fix the underlying problem and are of no help at all when you are developing a project that requires a SharePoint restart between compilations.

Before you continue reading you should realise that there are multiple factors that affect initial load time. There is the ‘loading of the assemblies and associated pre-compilation’ wait time, which can be solved by faster hardware, and then there is the problem outlined below, which may or may not be present in your environment. Read on to find out.

As it turns out this problem does not occur on all SharePoint installations, it only happens under a certain combination of circumstances. Fortunately I have identified the root cause as well as a solution.

Symptoms for STSADM:

• You start STSADM without any parameters

• There is a delay of about 30 seconds

• While you are waiting, and tearing your hair out because your deployment script has about 60 STSADM commands, there is no CPU activity, swapping or significant network traffic.

Symptoms for SharePoint sites:

• You make the first request of the day, or the first request after recycling the app pool because you are developing assemblies that sit in the GAC.

• There is a delay of about 2 minutes

• While you are waiting, and tearing your remaining hair out because you know you have to do this at least 50 times today, there is no CPU activity, swapping or significant network traffic.

So, what is going on here? Quite a few SharePoint developers are complaining about this on the Internet - and we even encountered the problem on the training disk images used by a renowned SharePoint training provider - but no-one has a real solution.

After yet another night of Googling around I found the solution in a posting about SQL Server 2005, which appears under certain circumstances to suffer from the same problem as SharePoint 2007. (Note that in my particular situation the problem is caused by SharePoint being slow, not SQL Server.)

The problem is that when loading signed assemblies the .net Framework checks the Internet based Certificate Revocation List. As our servers have, like most secure environments,  no outgoing connections to the public Internet, the connection to crl.microsoft.com times out after what appears to be 30 seconds. It probably does this a couple of times in succession, causing a 2 minute wait when spinning up SharePoint.

After the timeout the assembly is still loaded and the software works as expected, though very slow every time a new signed assembly is loaded for the first time, which happens a lot. The worst thing is that no entries are written to the event log and no exceptions are thrown so you are left completely in the dark about why your application is so bloody slow.

There are a couple of workarounds, which one works best is for you to decide:

1. Add crl.microsoft.com to your hosts file and point it to your local machine. Some people have reported success with this, but it didn’t work for me.

2. Allow your servers to directly connect to crl.microsoft.com. If your environment dictates the use of a proxy server, configure it using proxycfg.

3. Disable the CRL check by modifying the registry for all user accounts that use STSADM and all service accounts used by SharePoint. Find yourself a group policy wizard or run the vbscript at the end of this posting to help you out. Alternatively you can manually modify the registry for each account:

\[HKEY_USERS\\<userid>\\Software\\Microsoft\\Windows\\CurrentVersion\\WinTrust\\Trust Providers\\Software Publishing\]
"State"=dword:00023e00

4. Download the CRLs and add them to the server manually (I haven’t tested this, but it may work):

   1. Download: https://crl.microsoft.com/pki/crl/products/CodeSignPCA.crl

      https://crl.microsoft.com/pki/crl/products/CodeSignPCA2.crl

   2. Add them:

certutil -addstore CA CodeSignPCA.crl
certutil -addstore CA CodeSignPCA2.crl

5. Mark Russinovich has written a detailed post about this problem and provides a 5th option, which involves a manual change to each web application’s web.config file.

6. Brett Waring pointed me to an article that describes how to reduce the timeout using  a group policy.

We decided to go for Option 3 (disable CRL check) and life is good again, the improvement is significant.

VBScript to apply registry change:

The following script applies the registry change to all users on a server. This will solve the spin-up time for the service accounts, interactive users and new users.

const HKEY_USERS = &H80000003

strComputer = "."

Set objReg=GetObject("winmgmts:{impersonationLevel=impersonate}!\\\" \_

& strComputer & "\\root\\default:StdRegProv")

strKeyPath = ""

objReg.EnumKey HKEY_USERS, strKeyPath, arrSubKeys

strKeyPath = "\\Software\\Microsoft\\Windows\\CurrentVersion\\WinTrust\\Trust Providers\\Software Publishing"

For Each subkey In arrSubKeys

objReg.SetDWORDValue HKEY_USERS, subkey & strKeyPath, "State", 146944

Next

With thanks to Nik Shaw for the script.

Please read the comments section below, it contains some interesting replies including changes required for the various Windows versions as well as a PowerShell equivalent of the above mentioned VBScript  code.

Relevant links:

• Original article on the Microsoft Connect site that pointed me in the right direction.

• Microsoft KB article for resolving the problem in SQL 2005

• Proxycfg documentation

• SharePoint Warmup tool on CodePlex

• Andrew Connell’s warm up script

• Joel Oleson’s original warm up script