In one of my projects, the customer was upgrading from Windows Server 2003 (R2) to Windows Server 2008 (R2).
Since the customer requested of us to find out which applications were compatible with a x64 OS, we decided to use Microsoft’s Application Compatibility Toolkit for this.

After ACT 5.6 was installed, the next morning we received a XXX from the customer that there were enormous performance problems.
Where a users would take about 7 seconds to logon normally, now it would take 91 seconds.
And where starting Microsoft Office Word 2003 used to take about 4 seconds, now it would take 15.

So the troubleshooting began.
First we’ve checked if the collected data was written to the network storage we had configured and yes, the data was written to the location.
Second we’ve checked there were any conflicts with other applications by using procmon from Sysinternals, again cause was found.
Third, we’ve checked if there was an DiskIO/CPU/Memory problem on the servers or network storage but this was not the case.

So, I’ve started putting some exclusions in ACT (by editting ShimExclusions.csv).
First all executables from Microsoft Office were included, like for example winword.exe and excel.exe.
This did not resolve the performance problems, not even the part about starting the applications.
Secondly, I’ve included all components from the logon procedure of the Operating System and just to be thurrough I’ve also added the components from Citrix’s Presentation Server 4.5.
Now, the logon times were 72 seconds instead of 90… so it was an improvements but no nearly enough.

Next, I’ve started AutoRuns from Sysinternals and everything under the “Logon”, “WinLogon”, and “Services” tab I’ve also excluded from ACT.
Now, the logon times were brought back to 61 seconds, but keep in mind that the list of exclusions wrom ACT had 147 entries!

Since we were able to reproduce the performance problems without any problems (install ACT and logon) we had to start thinking beyond ACT.|
As always, first up was the Anti-Virus product this customer uses, which we later found out as McAfee.
We’ve added some exclusions in McAfee for our ACT, namely the process of ACT (actdcsvc.exe) and it’s entire installation directory (%ProgramFiles\Microsoft Agent Framework).

After some testing (including a reboot of the entire server farm and network storage) the performance issues remained.

So, back to one of my favorite tools… procmon.
I created a log with procmon and saw that McShield.exe and collector.exe were like fighting eachother.. what one touched, the other did to0…
So, in ACT I created an exclusion for “mcshield.exe” and in McAfee I’ve added the exclusion for “collector.exe”.

Result!
Logon times: 11 seconds.
Starting Word: 7 seconds.
Starting Excel: 7 seconds.

Strangely enough, we use ACT at a variaty of customers and none of them ever complained about anyhing like this.

So, just to be sure I “nuked” McAfee (renaming “mcshield.exe” to “mcshield.old” does the trick) and rebooted the server.
Guess what?
Logon times: 7 seconds.
Starting Word: 4 seconds.
Starting Excel: 4 seconds.

Since the customer has chosen for McAfee as their anti-virus solution, I’ve renamed “mcshield.old” back to “mcshield.exe”… namely because a terminal server ith internet access without an anti-virus solution is not an option for me.

Lesson: Configure all the right exclusions, and still something can cause negative performance.