Friday, August 29, 2008

Lingering Inter-Domain GC Objects

Recently, a customer experienced a very strange issue pertaining to inbound mail getting bounced from external sources intermittently with the error:

The following recipient(s) cannot be reached:
User on 6/11/2008 2:45 PM
A configuration error in the recipient's e-mail system prevented delivery of this message. Two recipients are configured with the same e-mail address. Contact your administrator.
<[Server] #5.1.4>

The Azaleos monitoring pulled the following from the event logs as well:

Alert: Event 9514 Sent By MSExchangeIS
Property: SourceName (MSExchangeIS)
Instance: Computer(MAILNODE1)
Two objects in the Directory have the same proxy - /DC=com/DC=Domain/DC=childdomain/OU=loc/OU=Departments/OU=dept1/CN=Last,First and /DC=com/DC=Domain/DC=childdomain/OU=loc/OU=Users/CN=Last,First.

At first glance this simply appears to be an issue with two legitimate users having the same email address, caused by replication issues, or creations / deletions at the same time. I searched AD on most of their Domain Controllers (DCs) for the SMTP address, and only turned up the one entry.

I decided to try to trick the system by removing that SMTP address from her account, and then sending the address email to find out where it went. When I did this I received:

First.Last@domain.com
The recipient's e-mail address was not found in the recipient's e-mail system. Microsoft Exchange will not try to redeliver this message for you. Please check the e-mail address and try resending this -message, or provide the following diagnostic text to your system administrator.

The intermittent portion of this issue tells me that it could very well only be a phantom object on a select number of DCs. I decided to run some scans using LDP, LDIFDE, and ADUC. The only difference this time would be to scan using port 3268, which is the Global Catalog (GC) port, rather than the LDAP 389 port.

I ran the following command to derive output against the various GCs in their domain:

ldifde -f C:\output.txt -t 3268 –s GC.domain.com -j C:\ -r "(ProxyAddresses=SMTP: First.Last@domain.com)"

Just the one user showed up again. At this point I remembered that in beginning, I was told that the user had moved between child companies of the parent not long ago. I ran the same scan against the other child domain, but still found nothing.

I ran message tracking to verify mail flow, and that mail always entered the organization the same way. The messages always come directly to the user’s new child domain as expected, and then get rejected. This tells me that it has to be a DC/GC on the new host site of the user (We will call this site A, and their original domain of employment a site B). I connected each DC/GC in site A via ADUC (Active Directory Users and Computers) on the GC port, to the site B domain to poll its records. A single GC had a duplicate record of the user!

Now that the problem has been found, how do we solve it? There are actually two problems. The first is that records viewed over the 3268 port are read only, and if you view the same thing over LDAP (389), you can’t see the record to delete it. Well Microsoft gives us a handy tool called repadmin that usually clears lingering objects like this up.

I ran “repadmin /showrepl” in order to display this server’s replication partners. After pulling a GUID of the DC/GC that I wanted to reference. After pulling the GUID, I inserted it into the following command:

repadmin /removelingeringobjects Problemserver.domain.com a130f9a9-42c5-4499-93d1-530f07fbbec7 DC=company,DC=domain,DC=com /ADVISORY_MODE

but I got the following error..

DsReplicaVerifyObjectsW() failed with status 50 (0x32):
The request is not supported.

This leads us to the second problem. The error occurred due to the problematic server being Windows Server 2000. This problem was unforeseen as every other DC in the organization is 2003. This specific case was resolved by the customers decommissioning the server as a GC because they had been intending to get rid of it for some time anyways. If they did not want to get rid of this server as a GC, however, we could have removed the lingering objects by following the instructions here:

http://support.microsoft.com/?kbid=910205

No comments: