Recently I had some unusual issues on a client site. We had an admin server that was running just the Exchange Management tools (EMC, EMS). Post installation of Exchange 2010 SP1 Rollup 3 we started getting performance counter errors in the event logs, about 15 errors at random points during the day. 

Exchange 2010
Source: MSExchange Common
EventID 106
Category (1) 
Performance counter updating error. Counter name is Total Admin Audit records saved, category name is MSExchange Admin Audit. Optional code: 2. Exception: The exception thrown is : System.InvalidOperationException: The requested Performance Counter is not a custom counter, it has to be initialized as ReadOnly.

Microsoft have various articles like http://support.microsoft.com/kb/982679/en-us which discuss various problems with their counters and the workaround is just to ignore them. Well that isn’t a very satisfactory answer when you have an event log full of red errors and a log management system alerting that the server is broken! 

Having done the usual Google search and found many people with various versions of our problem it turned out the fix wasn’t straight forward as we seemed to have been hit with multiple problems. 

Root cause was the performance counters were broken as everyone knows. We also were missing registry keys which meant not all of our PowerShell components were loading in the EMS. So in case anyone has this much fun with one of their servers, here are the fixes in order of running. 

Step 1. Fix your EMS PowerShell snapin to get all your commands back

When you open your Exchange Management Shell (EMS) your should be able to run Get-PSSnapIn and get the following appear:
Name : Microsoft.Exchange.Management.PowerShell.E2010
PSVersion : 1.0
Description : Admin Tasks for the Exchange Server 

Name : Microsoft.Exchange.Management.Powershell.Support
PSVersion : 1.0
Description : Support Tasks for the Exchange Server 

Name : Microsoft.Exchange.Management.PowerShell.Setup
PSVersion : 1.0
Description : Setup Tasks for the Exchange Server
Along with all of your standard PowerShell snapins. 

If any are missing, especially the Exchange Setup snapin, you won’t get very far. 

You need to edit the registry somewhat to fix this one.
Open Regedit and check out
[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\PowerShell\1\PowerShellSnapIns]
If you don’t see the following then you’ll need a bit of work done. 

To save anyone some work just copy and paste the following into a .Reg file. Make sure you update the ModuleName path to equal where you installed your Exchange files. 

Start —> 

Windows Registry Editor Version 5.00 

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\PowerShell\1\PowerShellSnapIns]
@=”" 

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\PowerShell\1\PowerShellSnapIns\Microsoft.Exchange.Management.PowerShell.E2010]
“CustomPSSnapInType”=”Microsoft.Exchange.Management.PowerShell.AdminPSSnapIn”
“ApplicationBase”=”D:\\Program Files\\Exchange Server\\bin”
“AssemblyName”=”Microsoft.Exchange.PowerShell.Configuration, Version=14.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35″
“Description”=”Admin Tasks for the Exchange Server”
“ModuleName”=”D:\\Program Files\\Exchange Server\\bin\\Microsoft.Exchange.PowerShell.Configuration.dll”
“PowerShellVersion”=”1.0″
“Vendor”=”Microsoft Corporation”
“Version”=”14.0.0.0″ 

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\PowerShell\1\PowerShellSnapIns\Microsoft.Exchange.Management.PowerShell.Setup]
“ApplicationBase”=”D:\\Program Files\\Exchange Server\\bin”
“Description”=”Setup Tasks for the Exchange Server”
“ModuleName”=”D:\\Program Files\\Exchange Server\\bin\\Microsoft.Exchange.PowerShell.Configuration.dll”
“PowerShellVersion”=”1.0″
“Vendor”=”Microsoft Corporation”
“Version”=”14.0.0.0″
“AssemblyName”=”Microsoft.Exchange.PowerShell.Configuration, Version=14.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35″
“CustomPSSnapInType”=”Microsoft.Exchange.Management.PowerShell.SetupPSSnapIn” 

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\PowerShell\1\PowerShellSnapIns\Microsoft.Exchange.Management.Powershell.Support]
“CustomPSSnapInType”=”Microsoft.Exchange.Management.Powershell.Support.SupportPSSnapIn”
“ApplicationBase”=”D:\\Program Files\\Exchange Server\\bin”
“AssemblyName”=”Microsoft.Exchange.Management.Powershell.Support, Version=14.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35″
“Description”=”Support Tasks for the Exchange Server”
“ModuleName”=”D:\\Program Files\\Exchange Server\\bin\\Microsoft.Exchange.Management.Powershell.Support.dll”
“PowerShellVersion”=”1.0″
“Vendor”=”Microsoft Corporation”
“Version”=”14.0.0.0″ 

–> End. Copy in between the markers, update your ModuleName to your path and import into your registry. 

So that is fix 1. This will get your EMS PowerShell session back up and running with the full range of Exchange 2010 Commands that you need. 

Step 2: Remove the bad Performance Keys.

For this we are going to open up a EMS PowerShell prompt and run the following. 

• Type add-pssnapin Microsoft.Exchange.Management.PowerShell.Setup and hit enter.
• Go to your Exchange BIN folder:  cd “D:\Program Files\Exchange Server\Bin\perf\AMD64″ (Change the path to where your perf folder is in your Exchange install)
• Type remove-perfcounters -definitionfilename .\eseperf.xml and press Enter.

You will now need to look very closely at the error in your event log 

As you can see we still had errors that were coming from the “MSExchange Admin Audit” performance counters. So far we had just removed the base ESE counters that covered most of our errors. 

• To fix the last of the problems you have to go to the very specific counter you have a problem with.
  CD “D:\Program Files\Exchange Server\Setup\Perf”
• We had to also run remove-perfcounters -definitionfilename .\AdminAuditPerfCounters.xml which then unloaded the last of our broken counters.
• Keep an eye on your logs and look at what Category the errors come from and just remove the specific sets from the Setup\Perf folder.

If you don’t plan on monitoring any Performance Counters on the server then you are done. As this was an Admin console we didn’t want any Exchange specific perf counters so we stopped here after we had cleared all of the error from the Application Log. 

If you are reading this and you have a Hub/CAS/Mailbox server with this issue then you will want to reload the counters now so that they are working. 

Step 3 (Optional): Fix the Performance Counters.

Back at your PowerShell prompt: 

• Type add-pssnapin Microsoft.Exchange.Management.PowerShell.Setup and hit enter.
• Go to your Exchange BIN folder: cd “D:\Program Files\Exchange Server\Bin\perf\AMD64″ (Change the path to where your perf folder is in your Exchange install)
• Type add-perfcounters -definitionfilename .\eseperf.xml and press Enter. This will reload the base ESE counters.
• CD “D:\Program Files\Exchange Server\Setup\Perf”
• Re-Add any other bad counters that you removed in the last step.
  add-perfcounters -definitionfilename .\AdminAuditPerfCounters.xml
• This will reload the counters and fix any permission errors in the registry that were stopping the keys being read. It will also replace any missing or broken keys from the various Service Packs and Rollups you’ve been applying.

If you are after specific counters then you can reload any that you need from the \Program Files\Microsoft\Exchange Server\V14\Setup\Perf directory in a default installation. 

I really hope this will fix any issues you have with Performance Counters on your Exchange servers and Exchange management boxes. A good admin will always investigate why there are ERRORS in event logs. Never ignore them as ignorance is rarely bliss in an IT environment  

Thanks,
Tim 

Tim Eichmann, CIO, Parative