Issues mounting NDMP Backed Up Luns

A coworker of mine cracked the case on this one with a bit of help from NetApp, but I thought I'd share as it was a strange little issue.  First off, what is an NDMP backup?

NDMP stands for Network Data Management Protocol, and was actually pioneered by Network Appliance (NetApp) in association with Intelliguard.  The purpose was to backup various platforms and provide interoperability.  Fast forward back to our current predicament..  

The current predicament is that someone uses Veritas to backup and restore.  They want to be able to restore from this, and have it correctly mount such that a they can recover data in Exchange.  The problem is that the restored lun won't mount in Snapdrive or otherwise.  Initially NetApp merely told us that the lun had to be in the root of the share rather than in a folder.. no problem.  Veritas must restore to a folder, but we can move it after the fact to the root. 

We then try to mount the lun, but it doesn't work.  It instead tells us that it is already mounted. 

The trick that my compatriot discovered was that it requires you to change the name as well by doing the following via command line:

lun move /vol/Volume1/restore.lun /vol/Volume1/restorenew.lun

He was then able to mount the lun.  Go team.

SMTP Categorizer Queue Length Spikes

The issue has cropped up with the Categorizer Queue Length intermittently spiking.  First off what is the Categorizer Queue Length?

This counter basically determines the items that reside in the categorizer queue.  The categorizer does a few things.  It resolves/validates recipients, determines whether the message should be queued for local or remote delivery, expands Distribution Lists (DLs), and detects limits and restrictions.

The first thing to check in order to resolve the spikes, would be that there are ample Global Catalog servers (GCs) and Domain Controllers (DCs) in the environment to perform the look-ups.  The best way to check this is on the DSAccess tab in ESM (2k3), or the System Settings tab under the properties of the Exchange server (2k7). 

It would also be prudent to run a network diagnostic tool in search of a bottleneck.

Microsoft also recommends monitoring the processor utilization of Inetinfo.exe (categorizer component) and using the e2kdsinteg config object from the ConfigDSInteg tool in order to check for malformed objects that could be slowing the process of directory look-ups.  You can get the tool here.

Knowing that the environment probably wasn't the shining star on the top of the Active Directory hill, this was the first thing to check.

The e2kdsinteg log came back with numerous entries for old mail servers, and objects not seen by human eyes in many moons.  My advice was to continue to run diagnostics for bottlenecks, and to investigate the purging of these rogue/malformed objects.

Outlook Anywhere Failing - RPC End Points - 6004

It was brought to my attention that autodiscover was not behaving correctly externally.  I ran it through Microsoft's Exchange connectivity tester @ http://www.testexchangeconnectivity.com/ and received the following output:

To resolve this first simple part I just went into the EMS and gave it an ExternalURL via:

Get-AutodiscoverVirtualDirectory | set-AutodiscoverVirtualDirectory -ExternalUrl https://autodiscover.domain.com/Autodiscover/Autodiscover.xml

I now received this error:

"Failed to ping RPC Endpoint 6004 (NSPI Proxy Interface)"

..and also RPC_S_SERVER_UNAVAILABLE error (0x6ba) was thrown by the RPC Runtime

Most curious about an RPC error at this level.  Perhaps a connection between the Hub/Cas and MBX server or MBX server and AD/DCs/GCs?  The environment was not 2008, nor was it using IPv6.

The following is what fixed my issue:

Using the configurations here I was able to remedy the situation.  Basically what happened was that it could not use DSPROXY via HTTP, and it is a known issue.  The fix is to:

1. Changes for Mailbox servers..

a. create a DWORD called "Do Not Refer HTTP to DSProxy" at HKLM\System\CCS\Services\MSExchangeSA\Parameters\ and the value set to 1.  This will, as it spells out, stop it from trying to use DSProxy when using HTTP.

b. HKLM\System\CCS\Services\MSExchangeSA \Parameters key "NSPI Target Server" to the FQDN of the domain controller that you would like used for profile creation.

2. Changes for Client Access Servers..

a. Ensure that the "PeriodicPollingMinutes" key at HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\MSExchangeServiceHost\RpcHttpConfigurator\ is set to zero.  This will ensure that the system won't continue to over write our settings every 15 minutes.

3. b. Also modify "ValidPorts" at HKLM\Software\Microsoft\RPC\RPCProxy such that it lists the DCs which can be accessed via port 6004.  An example of this would be:

domaincontroller.domain.com:6004;domaincontroller2.domain.com:6004

4. Changes for all Global Catalog (GC) servers..

a. Be sure that there is an REG_MULTI_SZ entry created named NSPI interface protocol sequences at HKLM\System\CurrentControlSet\Services\NTDS\Parameters\ and the value set to "ncacn_http:6004"

Testing autodiscover/Outlook anywhere now yields the following output in the connectivity tester:

You can double check these settings by configuring a profile in Outlook, then Ctrl+RightClicking the outlook icon on the system tray, and running "Test E-Mail Autoconfiguration."

For the full explanation I highly recommend reading the official blog post by Siddhartha Mathu at:

http://msexchangeteam.com/archive/2008/06/20/449053.aspx

Good read!