Exchange 2010 Set-WebServicesVirtualDirectory error: Unable to access the configuration system on the remote server

Problem:

When trying to run the Exchange powershell command Set-WebServicesVirtualDirectory the following error is returned:

Unable to access the configuration system on the remote server

Solution:

The fix is to re-register the application pool using the aspnet_regiis command (http://msdn.microsoft.com/en-us/library/k6h9cz8h(v=vs.140).aspx)

Identify if the application pool uses 32 bit or 64 bit by opening Task Manager, the w3wp.exe will show *32 next to the image name if the w3wp is running in 32-bit mode, if no *32 is shown next to the image name, then the worker process is running in 64-bit mode.

  • From the command prompt,
    • If its  V2.0 32 bit .Net 2.0 Framework then traverse to C:\windows\microsoft.net\framework\v2.0.50727
    • If its V2.0 64 bit .Net 2.0 Framework then traverse to C:\windows\microsoft.net\framework64\v2.0.50727
    • If its V4.0 32 bit .Net 2.0 Framework then traverse to C:\windows\microsoft.net\framework64\v4.0.30319
    • If its V4.0 64 bit .Net 2.0 Framework then traverse to C:\windows\microsoft.net\framework64\v4.0.30319
  • Then run the command aspnet_regiis -config+

Root Cause:

So why was it necessary to run this command at all? The Exchange Server was upgraded from 2008 R2 to 2012 R2. For several months Exchange ran fine post upgrade without any issue. The issue was only noticed when I tried to run the set-webservicesvirtualdirectory command.

Exchange 2010 New-MailboxExportRequest IncludeFolders Sub-Folders

To export all folders under a folder use following syntax:

new-mailboxexportrequest -mailbox [mailbox] `
-filepath [unc] -IncludeFolders "TestFolder/*"

If you try to export a sub folder, be sure to replace “/” with “//”, for example, if you want export a folder named “SubFolder”:

new-mailboxexportrequest -mailbox [mailbox] `
-filepath [unc] -IncludeFolders "TestFolder\\SubFolder"

And if you want to ensure you have the proper name of a folder to export, this command is handy:

Get-MailboxFolderStatistics -Identity [mailbox] |`
 ft identity

MS Exchange 2007 / 2010 Convert Mailbox to MailUser Powershell

NOTE: These commands (speficially #4) requires Powershell v2.0

1. Load the existing mailbox in to a variable

[PS] c:\scripts> $mailbox = get-mailbox -identity [username]

2. Disable the mailbox

[PS] c:\scripts> disable-mailbox $mailbox.samaccountname -confirm: $false

3. Create the MailUser

[PS] c:\scripts> Enable-MailUser -Identity $mailbox.userprincipalname -Alias $mailbox.samaccountname -PrimarySMTPAddress $mailbox.PrimarySmtpAddress

4. Update the MailUser with secondary addresses

[PS] c:\scripts> foreach ($i in ($mailbox.EmailAddresses)){if ($i.prefixstring -eq "smtp"){set-mailuser -identity $mailbox.userprincipalname -emailaddresses @{Add=$i.addressstring}}}

5. Update the MailUser with X500 addresses (if required)

[PS] c:\scripts> foreach ($i in ($mailbox.EmailAddresses)){if ($i.prefixstring -eq "X500"){set-mailuser -identity $mailbox.userprincipalname -emailaddresses @{Add="X500:"+$i.addressstring}}}

Emergency Exchange Server Recovery From Corrupted Database

Problem:

Two important events happened to me last week. Due to recent acquisitions I have been doing a large number of pst imports. The second is that we set up a new vmware cluster. We’ve been running low on system resources and the pst import has been going slow so my vmware admin advised to migrate the mailbox server to the new cluster. After moving the vm to the new cluster we increased the RAM and processor cores. Shortly thereafter I started the pst imports.

I started getting calls on the next business day that users were getting database disconnection messages in OWA. Sure enough on the server the 32GB of RAM and the 4 CPU’s were completely maxed out.

Using Process Explorer from Sysinternals we could clearly see that there was a private process memory leak on the Exchange database. A Private Process Memory Leak is what happens when a process tries to use all available system memory and we could see this happening with the store.exe process where even if the most minor process stopped, store.exe process would grow by a few additional KB.

At first blush this was a mailbox database corruption which would be a manageable problem however the system was completely unusable. We tried booting the server in to safe mode and even though that brought the CPU and Memory down you could not RDP to the server and interacting with the server at the console level was painstaking requiring several seconds inbetween mouse clicks. This made us think the problem might be with VMWare but more importantly we needed the email server back on line.

One other complication was that even though we were getting sucessful backup messages from our backup system the logs were not being truncated. There were 175,000 logs files! Yikes.

Resolution:

Our first thought was to copy out the database to a new server and perform an Exchange recovery and then repair the database. To get what files we needed to copy off we used ESEUTIL:

c:\windows\system32> eseutil /mh "D:\Exchange SErver\V14\Mailbox\[DatabaseFolder]\[DatabaseFile].edb

This command provides some really useful output, most importantly the the list of log files needed to perform a database repair.

Extensible Storage Engine Utilities for Microsoft(R) Exchange Server
Version 14.02
Copyright (C) Microsoft Corporation. All Rights Reserved.
Initiating FILE DUMP mode...
         Database: D:\Exchange Server\V14\Mailbox\MSX06MBXDB01\MSX06MBXDB01.edb

DATABASE HEADER:
Checksum Information:
Expected Checksum: 0x30bc20b7
  Actual Checksum: 0x30bc20b7

Fields:
        File Type: Database
         Checksum: 0x30bc20b7
   Format ulMagic: 0x89abcdef
   Engine ulMagic: 0x89abcdef
 Format ulVersion: 0x620,17
 Engine ulVersion: 0x620,17
Created ulVersion: 0x620,17
     DB Signature: Create time:08/15/2012 14:39:14 Rand:14004066 Computer:
         cbDbPage: 32768
           dbtime: 799434390 (0x2fa66696)
            State: Dirty Shutdown
     Log Required: 587149-587170 (0x8f58d-0x8f5a2)
    Log Committed: 0-587170 (0x0-0x8f5a2)
   Log Recovering: 0 (0x0)
  GenMax Creation: 09/19/2012 09:08:06
         Shadowed: Yes
       Last Objid: 730388
     Scrub Dbtime: 0 (0x0)
       Scrub Date: 00/00/1900 00:00:00
     Repair Count: 0
      Repair Date: 00/00/1900 00:00:00
 Old Repair Count: 0
  Last Consistent: (0x8EE72,A,54)  09/18/2012 16:08:25
      Last Attach: (0x8EE73,9,86)  09/18/2012 16:08:25
      Last Detach: (0x0,0,0)  00/00/1900 00:00:00
             Dbid: 1
    Log Signature: Create time:08/15/2012 14:39:13 Rand:14018268 Computer:
       OS Version: (6.1.7601 SP 1 NLS ffffffff.ffffffff)

So based on the log required field I was able to shorten the list of log files I needed to 1,300 files. Sadly I couldn’t copy them off again because of the performance issues. So we thought to dettach the storage volumes from the existing volumes and attach to the new server. Again we had a problem. In the VMWare console, even with the vm turned off, the Remove button when highlighting the drives of the vm were greyed out. At this point we had something to go to VMWare support with and say that there was a problem with VMWare.

To quote the VMWare support engineer, “That’s really wierd”, the solution was to remove the vm from inventory and then remove the host from inventory and rejoin both.

Now we had the storage volumes attached to the new server which put us on the road to getting operational again. The steps we followed were:

  1. Install Windows Server with the same Server Name and same OS configuration. (Disconnect “source” machine from the network.)
  2. Install all the prerequisites for exchange 2010 SP2 http://technet.microsoft.com/en-us/library/bb691354.aspx
  3. Install Exchange Server with /recoverserver switch having Exchange 2010 SP2 binaries.http://technet.microsoft.com/en-us/library/dd876880.aspx
  4. Copy the old database to the temp location. Same to the drive which will contain the database in future. (In our case we attached the drives.)
  5. Run soft recovery and bring the database in clean shutdown state.
    1. Before running the soft recovery just to be sure we had all the log files we ran the command to verify the log files:
      1. C:\Windows\system32>eseutil /ml F:\Recovery\E01
    2. Run soft recovery command:
      1. C:\Windows\system32>eseutil /r E01 /D "[DriveLetter]:\[PathToMailboxDBFolder]\[DatabaseFile].edb" /L "[DriveLetter]:\[LogFolder]"
    3. *** It’s also important to note when recovering an Exchange database you need the very first log file, commonly named E01.log and there should be no .jst files in the directory.
  6. Mount the database and monitor performance with the EXMon.exe tool http://technet.microsoft.com/en-us/library/bb508855(v=exchg.65).aspx
  7. Of course there were some mailboxes that were trying to use up 100% of the processor like we expected so we ran the on-line mailbox repair on the database using powershell:
    1. [PS] C:\Windows\system32>New-MailboxRepairRequest -database MSX06MBXDB01

After running mailbox repair (which took several hours) the server was back to normal and everybody had access to their mailboxes again.

So there you have 3 days of work in a bite-sized package.