Exchange 2016 Hybrid : TLS negotiation failed with error UnknownCredenta


I was adding couple of Exchange 2016 servers with CU2 to the Hybrid configuration wizard to send and receive emails to Exchange Online. On Exchange Online Admin center, I configured the receive connector to Office 365 o verify the subject name on the certificate for TLS authentication.

The problem is that emails are not being sent to Office 365 via the send connector. After enabling the protocol logging on my Exchange 2016 hybrid servers [Get-SendConnector “outbound to Office 365” |Set-SendConnector -ProtocolLoggingLevel verbose] , and opening the smtpsend log file, I can see many TLS failures:

016-07-19T12:13:14.863Z,Outbound to Office 365,08D3AFC581A92DD3,3,,,>,EHLO,
2016-07-19T12:13:14.910Z,Outbound to Office 365,08D3AFC581A92DD4,2,,,<,”220 Microsoft ESMTP MAIL Service ready at Tue, 19 Jul 2016 12:13:14 +0000″,
2016-07-19T12:13:14.910Z,Outbound to Office 365,08D3AFC581A92DD4,3,,,>,EHLO,
2016-07-19T12:13:15.004Z,Outbound to Office 365,08D3AFC581A92DD3,4,,,<,250 Hello [] SIZE 157286400 PIPELINING DSN ENHANCEDSTATUSCODES STARTTLS 8BITMIME BINARYMIME CHUNKING,
2016-07-19T12:13:15.004Z,Outbound to Office 365,08D3AFC581A92DD3,5,,,>,STARTTLS,
2016-07-19T12:13:15.051Z,Outbound to Office 365,08D3AFC581A92DD4,4,,,<,250 Hello [] SIZE 157286400 PIPELINING DSN ENHANCEDSTATUSCODES STARTTLS 8BITMIME BINARYMIME CHUNKING,
2016-07-19T12:13:15.051Z,Outbound to Office 365,08D3AFC581A92DD4,5,,,>,STARTTLS,
2016-07-19T12:13:15.145Z,Outbound to Office 365,08D3AFC581A92DD3,6,,,<,220 2.0.0 SMTP server ready,
2016-07-19T12:13:15.145Z,Outbound to Office 365,08D3AFC581A92DD3,7,,,*,” CN=*, OU=IT, O=contoso International (L.L.C), L=Dubai, S=Dubai, C=AE CN=thawte SHA256 SSL CA, O=””thawte, Inc.””, C=US 0D92CFF6070B73AD5722EC8B4DA3389B AAA3D3DADA6891A2CCB3134D0B2D7764F1351BC4 *”,Sending certificate Certificate subject Certificate issuer name Certificate serial number Certificate thumbprint Certificate subject alternate names
2016-07-19T12:13:15.145Z,Outbound to Office 365,08D3AFC581A92DD3,8,,,*,,TLS negotiation failed with error UnknownCredentials

I am sure the certificate is fine as the other hybrid servers are using the same certificate and they are able to send emails to Office 365. Also on the event viewer, I am seeing the following error:

TLS Error Office 365 Exchange Hybrid


So finally, I tried something and it worked. I opened the certificate store, and I was checking the permissions on my certificate private key, the certificate I am using for the TLS connection.

TLS Error Office 365 Exchange Hybrid2

I can see the following permissions on the private key:

TLS Error Office 365 Exchange Hybrid3


So I added the Network Service and I gave it READ access. After that everything worked just fine. Try to give EVERYONE Read access if things are not working yet.

Hope this will help someone, leave a note if it did 🙂

Office 365 and Group Moderation Tips

In the process of testing out Office 365 and Exchange hybrid configuration, an interesting thing happened that I want to share with you.

I have an on-premise Exchange 2010 implementation and couple of users are hosted at Office 365. All hybrid configurations are set and connectors are configured to route emails between the two spaces.

Everything is working fine, and mailboxes hosted on Office 365 are working just fine. Things started to get interesting when people start to send emails to moderated distribution groups.

When someone sends email to a moderated groups, and the moderator is hosted on Office 365, the buttons for Approve and Reject are not showing at his email client.

It turned out that a setting called TNEF (Transport Neutral Encapsulation Format) is causing this to happen. We need to make sure TNEF format is enabled when sending emails out to Office 365 tenant.

The TNEF setting is configurable per remote domain (Get-RemoteDomain) and (Set-RemoteDomain).

By default, there is a default RemoteDomain configured in your Exchange environment called (Default). If you hit (Get-RemoteDomain), you will see all settings that controls the behavior of email communications and format when sending emails to external parties. One of the settings is TNEFEnabled.

Now that we have Office 365 hybrid setup, the HCW creates for us a remote domain in the on-premise organization to allow TNEF (

That is great. So all what we need to do is to configure that remote domain (Set-RemoteDomain -TNEFEnabled ….) and all is done, right?

There is a small thing left to say. When Office 365 sends emails regarding moderated groups, the messages come from a system mailbox in the tenant with email address SystemMailbox{bb558c35-97f1-4cb9-8ff7-d53741dc928c} So we need to add a new remote domain called

So let us start typing some PowerShell commands:

New-RemoteDomain -Name “Hybrid Domain –” -DomainName

Now we have two remote domains:


We have then to configure both remote domains to allow TNEF format. I also recommend configuring many other settings on the way.

Set-RemoteDomain -Name “Hybrid*” -IsInternal $true -TargetDeliveryDomain $true -AllowedOOFType InternalLegacy -MeetingForwardNotificationEnabled $true -TrustedMailOutboundEnabled $true -TrustedMailInboundEnabled $true -UseSimpleDisplayName $true -TNEFEnabled $true

That’s great. Now we have configured both remote domains to enable TNEF format. I have also noticed that when on premise mailboxes try to communicate with the Office 365 system mailbox for moderation actions (Approve,Reject), they are receiving authentication errors. To fix that, add the to the address space of the Office 365 connector:

Set-SendConnector “Outbound to Office 365″ -AddressSpaces @{Add=””}

Finally, it makes sense to instruct the Office 365 tenant to treat the on premise Exchange organization the same way. Suppose my on premise domain domain is, then connect to your office 365 Exchange PowerShell, and type:

New-RemoteDomain -Name “Hybrid Domain –” -DomainName

Set-RemoteDomain -Name “Hybrid*” -IsInternal $true -TargetDeliveryDomain $true -AllowedOOFType InternalLegacy -MeetingForwardNotificationEnabled $true -TrustedMailOutboundEnabled $true -TrustedMailInboundEnabled $true -UseSimpleDisplayName $true -TNEFEnabled $true.

Reference Link 

Exchange Email Moderation Super Cool Script – Must Have

Hi everyone,

Email Moderation is one of my best features in mail flow restrictions in Exchange. You can assign one or more moderators to groups, so that if any one of them approves the email being sent, then this will release the email to that moderated group.

You can view email moderation information from Exchange GUI admin tools, but for dynamic moderated groups, you shall use PowerShell to view and configure email moderation. Usually, dynamic groups that has country or office filter criteria will contain lots of people and you want them to be moderated.

No Dashboard for moderation info

The first issue people have with email moderation is how to get a report with all email moderated groups and their moderators, and bypass moderation recipients. There is no dashboard that shows all this information in one place.

Disabled or Orphan Moderators

The second issue that Email Administrators will face is moderation list maintenance. Suppose that GroupA has one moderator called John. John decided to leave the company, and his account is now disabled. Now GroupA has no moderators. It has Moderation status set to true, but no moderators. Some cleanup job need to be performed frequently to check for the health and existence of the moderators.

Single Moderator Issue

Moreover, the best recommendation is to have at least two moderators for each moderated group, so that if one of them is not available or on a leave, the other one can moderate that group. You may want to have a regular checks to detect moderated groups with one moderators only.


I have created a PowerShell Script that you can run, and it will do the following:

  1. Generate CSV file that lists all moderated groups in your environment with the following Info:
    1. Group Name
    2. Dynamic mailing group or not
    3. Moderators list
    4. Bypass moderation list
    5. Managed By list.
    6. Email Address
    7. Alert column if a single moderator is detected.
    8. Health Field to indicate if one of the moderators is disabled or does not have mailbox anymore.
    9. Empty Moderator List warning
  2. Three Log Files will be generated. One for information, one for empty moderator groups, and one listing groups with disabled mailbox moderators.


Download the Script here

You can download the script from here Get-CorpModerationInfo

Exchange Online EOP and Send of Behalf

Some one asked me recently about an interesting scenario on which emails are send on behalf of another party, and how Exchange Online Protection (EOP) will act in this case.

There are two FROM values in the SMTP world:

  • RFC 5321  (MailFrom)
  • RFC 5322 (From)

Outlook displays the RDC 5322 From address to end users, and this is the address that is used in the user’s safe sender list.

EOP inspects both values for blocked and allowed senders and domains. Exchange Online Protection EOP and outlook handle safe sender lists differently.

In most cases, those two values are the same which is normal. Things become interesting when someone is sending emails on behalf of another party. Let us take a simple example:

Mailfrom vs MAIL RFC

  • Contoso corporation is trying to send email to their customers and they contracted with third party to send their news emails.
  • The contractor company sends the news email on behalf of contoso.
  • The email that was sent has the following values:
    • RFC 5321 MailFrom:
    • RFC 5322 From:
  • One of the customers who are using EOP receives the news email, and in Outlook, he can see that the sender is
  • The user added this address to the safe list senders.
  • Because EOP inspect both RFC from addresses, the next time an email was sent by the contractor, EOP will white-list that email respecting the user’s safe list.

Usually the RFC5321 address is the one used by EOP to do SPF checks and send NDR or bounced messages.

Reference article:

Exchange 2013 Certificate Revocation Failed

Hi everyone,

I want to share with you my personal experience in troubleshooting an interesting problem where Exchange 2013 management interface shows the status of a certificate that I had imported as (Revocation Status Failed).

So why this is happening? When Exchange 2013 tries to enumerate certificates on the computer store for you in the Exchange Admin Center, it will try to check the revocation status for each certificate to make sure the certificate is Valid. To do that, it will try to download the CRL (Certificate Revocation List) file from the internet by looking at the certificate  (CRL Distribution Points) attribute of that certificate.

CRL Certificate Exch2013

This CRL file download is happening in the background when the server is restarted and using the SYSTEM account. So the SYSTEM account is trying to download something from the internet in the background, and for sure it will use the proxy settings in the IE that is configured for SYSTEM account, which is auto detect proxy settings.

Since the server is not configured to use DHCP, then the auto discover process will go to DNS and search for ,  for example (, and since I have such record in my DNS pointing to my proxy, then the SYSTEM account is trying to connect to my proxy, perhaps authenticate and then tries to download the CRL file.

This means also that each time the SYSTEM account in the Exchange 2013 needs to connect to internet, it will do that via my proxy which is something I do not like. I would rather like to have a direct connection from Exchange 2013 to the internet, especially if we are talking about hybrid configuration and Office 365.

How to solve this issue?

I started to think, if i could log on to the computer using SYSTEM account, open the IE and remove the Auto-detect proxy setting, then the problem would be solved and i will have a direct internet connectivity that will eliminate any complexity or authentication requirements on my proxy.

So i went to one of my favorite sites [Windows SysInternals] ,  and i have downloaded the PsExec tool, and copied it to the C:\ drive of my Exchange server. This tool has the option to initiate an executable remotely or locally using local system account.

The idea is that I want to run CMD using SYSTEM account interactively and then open IE from there. Once IE is opened in front of me using SYSTEM account, i can then remove the proxy auto-detect chec kbox from there. To do that, I logged on as a local administrator to one of my Exchange 2013 where i have PsExec copied on the C drive, and then I run:

psexec -i -d -s cmd

CRL Certificate Exch2013 2

This will open a new CMD window for me. From that window, I can type WhoAmI and I can see that the CMD window is running under the SYSTEM account.

CRL Certificate Exch2013 3

Now, I will open IE using SYSTEM context.

CRL Certificate Exch2013 4

and from there I will remove the auto-detect proxy settings, so that SYSTEM will not use proxy when connecting to the internet to fetch the CRL of my certificate.

CRL Certificate Exch2013 5

Exchange Dashboard Email Report – New version available v2.4.9

Hi everyone,

I am so glad that many people are downloading and using the Get-CorpEmailReport in their networks to get a full overview of their Exchange Environment. The script has two main features:

  1. It works with all version of Exchange [Up to Ex2013].
  2. It reports Exchange Office 365 users.

I have updated the script from version 2.4.8 to 2.4.9 and i have included couple of fixes:

  • Bug fixes in loading Exchange Management Shell if the Exchange is installed in a drive other than C:\.
  • Bug fixes in loading Exchange Management Shell on Exchange 2013.
  • Now the script supports and report on latest Exchange 2013 version.

You can read more about the script here, and you can download it directly from Microsoft Script Gallery here.

Charts with PowerShell 5

Charts with PowerShell 3


Lync and Exchange Web Service Integration When Using Different Domain [Updated March 2017]

If you are have Microsoft Exchange and Microsoft Lync, then you may find this post interesting. It is about the Lync integration with Exchange Web Services EWS.

Company A:

  • AD Domain : CONTOSO.COM
  • Exchange with SMTP domain : CONTOSO.COM
  • Lync with SIP domain : CONTOSO.COM
  • Split DNS configuration.

Company A acquired a small company and they migrate them fully to their domain. Nevertheless, a couple of people wanted to have as their primary SMTP address for business need.

Now people with as their primary SMTP address, are experiencing strange and broken behavior between their Lync 2013 client, and Exchange web services. People with as their primary SMTP address, still using CONTOSO\username logons, and CONTOSO.COM as their SIP domain.



Adding TrustModelData Registry Key with value ( to the machines with Lync 2013 client that are experiencing the problem.

The registry key can be applied on a machine level or user level (See the TechNet article):

  • “HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Office\15.0\Lync\TrustModelData” “HKEY_CURRENT_USER\SOFTWARE\Policies\Microsoft\Office\15.0\Lync\TrustModelData”

There is a group policy to configure this also in the admx/adml files for Office 2013 .  This group policy setting called (Trusted Domain List) and it is mentioned here in this TechNet Article.

Exchange 2013 Capacity Planning

If you are migrating or planing to deploy Exchange 2013, you may want to know about what are the small changes that affect your capacity planning. This post will help you quickly get an overview about those changes, in order to do a more accurate capacity planing.

When you size Exchange 2013, you are sizing for three roles:

  • Mailbox Role : consolidates most Exchange components.
  • CAS Role : is an effective stateless proxy.
  • Active Directory : Exchange depends heavily in AD and you should have enough hardware for AD to support this heavy dependency.

Point 1: More free disk space on the install drive

Exchange 2013 contains built-in performance motioning components: Exchange Diagnostics Services (EDS) that collects performance data to be used by Microsoft engineers in case you open a support case, capacity planning, sizing guidance, or for performance bug detection. This is enabled by default.

There are also many other logs that are enabled by default. All of this requires lots of space on the install drive of Exchange 2013 server. A minimum of 30 GB Free space is required on the install drive.

Point 2

Capacity planning process for Exchange 2013 can be divided to three stages:

  • Read the capacity guidance.
  • Collect User Profiles and Average Message Size
  • Define constraints based on requirements
  • Input profile data and design constraints to the calculator
  • Review the output and consider the impact of each option
  • Finalize the design and do documentation.

Exchange 2013 capacity planning



Exchange 2013 capacity planning 2


Point 3 : Exchange 2013 targets balance use of hardware

  •  Rather than having set of roles in the product that use hardware in different ways and that not necessary use all the hardware on the server in the best way possible, instead now we have smaller number of roles that ideally use all the hardware that is available to that role in a balanced way.
  • Roles are loosely coupled and scale independently.

The whole idea here is that the Mailbox Role now consolidates most Exchange components, and Microsoft is pushing towards consolidating CAS Role with Mailbox Role ( also called Building Block Architecture). In this way, all the hardware available to the server will be used more efficiently.

Point 4: Memory requirements have increased in Exchange 2013

Exchange 2013 capacity planning 3


Point 5: New Mailbox Role

Most of Exchange 2013 components are now hosted in the new Mailbox role. Microsoft  also recommends to collocate the CAS role with the Mailbox role to utilize hardware resources even better.

The new Mailbox role provides simplified deployment and connectivity model. Less roles to provision and worry about, and less network packets between the old multi roles Exchange servers.

The new Mailbox role also provides  balance resource utilization and hardware efficiency, because instead of spreading roles between different servers, roles are now consolidated.

Cache effectiveness:  In previous Exchange architectures, processing and email traffic for a particular user could occure on many servers through the topology, so if i am opening my email from outlook and ActiveSync, then i may have two CAS servers processing my request. So my cached data stored on the servers would become useless as soon as those connections moved to other servers. In Exchange 2013, all workload processing for a given user occurs on the Mailbox server hosting the active copy of that user’s mailbox. Therefore, cache utilization is much more effective.

Point 6: New CAS Role

The new CAS role is now completely stateless proxy from a user perspective, so it becomes very easy to scale up and down as demands change by simply adding or removing servers from the topology. Compared to the CAS role in prior releases, hardware utilization is dramatically reduced meaning that fewer CAS role machines will be required.

Point 6: Storage Capacity

You always size

  1. Mailboxes
  2. Logs
  3. Indexes

In previous Exchange architectures, we always add 20% from the database size, as database overhead. In Exchange 2013, this overhead is now 0%

On the other hand, Content Index (CI) size is now 20% of the EDB + space for additional index set per volume (use for master merge maintenance process).

So if you have multiple databases per volume, you will have only one additional index log set. So by having multiple DBs per volume, you will save space that would other wise be consumed by the master merge log set. The master merge indexes are computed as 20% of one of the databases on the volume.

In Summary, you need to calculate index space as the following:

  1. Content index = 20% from the database size.
  2. Master Merge logs : one set per volume =  20% of the average size of the databases on that volume.

 Point 7 : Background Database Maintenance

Storage bandwidth is about things crossing between the server and the storage system. Bottlenecks were caused by the Background Database Maintenance (BDM), and in Exchange 2013, BDM is now consuming 1MB/sec/DB copy, significant reduction from 2010.

Point 7: Unified Messaging

Voice mail transaction is a heavy consumer of CPU and now the UM is part of the mailbox role now. If a server is CPU starved, then the voice mail transcription may be skipped while the voice mail is delivered.

Import from PST to online archive ? You need PowerShell Statistics Report



Well, say you have a project to import PST files to online archives in your Exchange environment. You start importing PST files to online archives one by one, and you need a way to monitor the progress and see which servers are doing the move and how long will it take to finish the current import operations. At the end, you will receive a nicely formatted HTML table to your email address.




Items Reported

This script will gather information about all mailbox import operations that are in progress (status = InProgress) and will report the following:

– User Name
– User SamAccount Name
– Office
– Targeet Database
– Percent Complete
– Queued Time Stamp
– Start Time Stamp
– Last Update Time Stamp
– Overall Duration
– Bytes Transferred
– Target Root Folder
– CAS Server doing the import process


Download the script now

You can download the script here : Get-CorpMailboxImport


Generate the HTML report with SMTP Email option
\Get-CorpMailboxImport.ps1   -MailFrom   -MailTo   -MailServer

Exchange Multi-Mailbox Search – Segregation of duties


The security or legal team needs access to search corporate mailboxes for keywords in order to investigate a security or legal incident.

Giving that person the ability to view and access other mailboxes without proper auditing is something most organization fear to do, even if that person is trusted and is a senior person.

Microsoft Exchange platform starting from Exchange 2010 I guess, comes with a new feature called Multi Mailbox Search . The problem with giving a person the ability to do searching on corporate mailboxes is still the same.

How Multi Mailbox Search works

I will not go through the details of how this feature works, as you can read on TechNet about it. Instead I will highlight couple of points:

Exchange 2010 introduces the Discovery Management Role and the Discovery Search Mailbox.  By default no users are members of this role and the user associated with the Discovery Search Mailbox is disabled and it cannot receive e-mail.

  • You start by granting a domain user “John” the role of Discovery Management in Exchange by running:

Add-RoleGroupMember -Identity “Discovery Management” -Member John

  • Then John can go to his Outlook Web App > Exchange Control Panel, and he will have access to the Reporting section under My Organization

Multi Mailbox Search

  • From there John can specify a search criteria as shown below.

Multi Mailbox Search 2

  • The results of the search will be sent to the built in system mailbox called (Discovery Search Mailbox).

John is granted automatically access to that (Discovery Search Mailbox) where he can view the results. This is because the (Discovery Search Mailbox) is configured by default with (contoso\Discovery Management) group having Full Mailbox Access. John is added automatically to that group once he is granted the “Discovery Management” Exchange Role previously.

Note: The problem with this approach is that John can perform any search or mailbox discovery on corporate mailboxes without proper control or auditing and this is extremely something to worry about.


The solution is simply a segregation of duties, where one person performs the search and other person gets access to view the result.

In this scenario, John can only go to his OWA experience and perform the multi-mailbox search with any criteria he wants, and the results will be sent to the (Discovery Search Mailbox). John should not have access to that system mailbox, and thus cannot view the results of his own search.

Now, Sue is another security administrator and she is granted full mailbox access to the (Discovery Search Mailbox) and she can see the result of the multi-mailbox search performed by John. This means that one person can do the search and cannot view the results, where the other person can view the results but cannot do the search. In other words, we require two different people to act in order to do such multi-mailbox search on corporate mailboxes.

How to do it:

  • For John, we will add him to the “Discovery Management” Exchange Role

Add-RoleGroupMember -Identity “Discovery Management” -Member John

  • For Sue, go to Exchange Management Console, search for “Discovery Search Mailbox”, right click and click “Manage Full Access Permission” and do the following:
    • Remove CONTOSO\Discovery Management
    • Add CONTOSO\Sue

Multi Mailbox Search 3

  • Ask John to do the multi-mailbox search from his OWA experience
  • Once done, the results are sent to the “Discovery Search Mailbox”, and John cannot view it although he is member of the (Discovery Management) role, but he cannot access it as we removed the full mailbox access from that mailbox for the AD security group “Discovery Management”.
  • Now John will call Sue and asks her to access that discovery mailbox by typing:

Note: you can get the discovery mailbox SMTP. You can figure out this SMTP by searching for the “Discovery Search Mailbox” in the Exchange Management Console and view the SMTP address from there.

Multi Mailbox Search 5

Exchange 2010 DR (Disaster Recovery and Data Center Switch Over) – Unleashed P5

This guide simply explains in a very easy way, all the technologies and procedures that you need to know to perform Exchange 2010 data center switch over, recovering DAG member or stretching DAG between sites.

Check other parts:

Exchange 2010 DR (Disaster Recovery and Data Center Switch Over) – Unleashed P1

Exchange 2010 DR (Disaster Recovery and Data Center Switch Over) – Unleashed P2

Exchange 2010 DR (Disaster Recovery and Data Center Switch Over) – Unleashed P3

Exchange 2010 DR (Disaster Recovery and Data Center Switch Over) – Unleashed P4

 Quick Tricks when switching over a secondary Datacenter: Activating CAS Servers

If the primary datacenter has the following URLs internally and externally

  •  (Outlook Anywhere)
  • (Outlook Web Access)
  • (Exchange ActiveSync)

And the secondary site has:


And suppose SCP for Autodiscover for CAS servers in the primary datacenter points to where SCP for CAS servers in the secondary datacenter points to Suppose also that the public points externally to primary datacenter publishing rule

During Data center Switch Over:

  1. OWA :

Change the IP address for to point to in the internal and external DNS servers. This really depends if the primary data center will be off for long time.

You can also chose not to change this DNS name if the primary CAS servers are online since they will do the redirection.

  1. EAS :

Change the IP of to point to in the internal or external DNS servers. You can also chose to tell the users to manually change this manually on their mobiles.

  1. Outlook Anywhere :
    • Either manually let users to change their outlook proxy settings to
    • Automatic solution would be making sure Autodiscover service is reachable internally and externally so that outlook profile repair will do the trick and switch to
    • NOTE: VERY IMPORTANT: Don’t ever try to change the DNS name of to point to This will always fail as the subject name of the certificate in LON datacenter is while the proxy settings in user outlook profile is

Exchange 2010 DR (Disaster Recovery and Data Center Switch Over) – Unleashed P4

This guide simply explains in a very easy way, all the technologies and procedures that you need to know to perform Exchange 2010 data center switch over, recovering DAG member or stretching DAG between sites.

Check other parts:

Exchange 2010 DR (Disaster Recovery and Data Center Switch Over) – Unleashed P1

Exchange 2010 DR (Disaster Recovery and Data Center Switch Over) – Unleashed P2

Exchange 2010 DR (Disaster Recovery and Data Center Switch Over) – Unleashed P3

Exchange 2010 DR (Disaster Recovery and Data Center Switch Over) – Unleashed P5

Restoring Services in the Primary Datacenter

  1. Power on the primary mailbox servers. If you open the cluster console on them, you can see that they reflect that they are evicted from cluster. Database copies on them are marked as Failed and there is no way to mount them on primary servers.


Verify that Cluster service on the DAG members in the primary datacenter have a startup type of DISABLED. If they do not, either the Stop-DatabaseAvailabilityGroup command was not successful or the DAG members in the primary datacenter failed to receive eviction notification after network connectivity between datacenters was restored. Do not proceed until Cluster service cleanup has occurred and Cluster service has a startup type of DISABLED. You can optionally run the following command on the DAG members in the primary datacenter to forcibly cleanup the outdated cluster information: “Cluster node /forcecleanup”

  1.  Run the Start-DatabaseavailabiltyGroup –Identity DAG1 –ActiveDirectorySite NYC command on them.

Note that powering those servers in the primary site will not be risky as they are out of DAG configuration. The start-DatabaseAvailabilityGroup command will return them to the DAG again.

 Also remember that we have performed the Move-ActiveMailboxDatabase command during switchover to be servers in the secondary site. That’s why when you start-DatabaseAvailabilityGroup on primary servers, they will notice that the databases are active on secondary mailbox servers and will not try to do anything.

After running this Start command, the primary mailbox servers will start appearing in the cluster console as cluster nodes functioning normally.


  1. Run Set-DatabaseAvailabilityGroup cmd without any parameter to make sure the right Quorum mode is being used. This command also will seed all changes on the passive copies.
  2. Database copies on the primary site will start seeding automatically and will turn healthy eventually.
  3. Leave the database to replicate over time and sync from Secondary datacenter to Primary. Then proceed to the below steps.
  4. Note that the DAG is using the alternative witness server. In order to use a witness server in the primary site, and if you still have the old witness server, then use Set-DatabaseAvailabilityGroup -Identity DAG1 command. If we want to assign new witness on the primary datacenter, then add the witness parameters to the previous command.
  5. Notice that the default cluster group is hosted on the secondary site which means that the Primary Active Manager PAM is located on the node who holds the default cluster group.

To identify the PAM server, run: Get-DatabaseAvailabiliyGroup –Identity DAG1 –Status |FL *Primary*

  1. You can move the default cluster group to the primary mailbox server by running Cluster group “Cluster Group” /MoveTo:EX01.
  2. Dismount databases in the secondary datacenters and move the CAS URLs.
  3. After DNS is replicated and the cache is refreshed, use the Move-ActiveMailboxDatabase for the copies in the primary site.
  4. Mount database copies in the primary site.
  5. Outlook clients will find a message to indicate that the administrator has changed something and the outlook need to be restarted.


Note : When mounting database copies on the primary site, sometimes you will face issues like database cannot mount because index problem. For this scenario, you can run :

Update-MailboxDatabaseCopy DBName\FailedToMountServer –CatalogOnly

If this didn’t work, use:

Move-ActiveMailboxDatabase “Database Name” -ActivateOnServer DestinataionServer SkipClientExperienceChecks

Note that this command is powerful, look at this :

Move-ActiveMailboxDatabase “Database Name” –ActivateOnServer –Options

Exchange 2010 DR (Disaster Recovery and Data Center Switch Over) – Unleashed P3

This guide simply explains in a very easy way, all the technologies and procedures that you need to know to perform Exchange 2010 data center switch over, recovering DAG member or stretching DAG between sites.

Check other parts:

Exchange 2010 DR (Disaster Recovery and Data Center Switch Over) – Unleashed P1

Exchange 2010 DR (Disaster Recovery and Data Center Switch Over) – Unleashed P2

Exchange 2010 DR (Disaster Recovery and Data Center Switch Over) – Unleashed P4

Exchange 2010 DR (Disaster Recovery and Data Center Switch Over) – Unleashed P5

3. Data Center Switch Over

3.1 Terminate the primary data center

  • DAG Members in the primary data center must be marked as stopped. Stopped is the status of Active manager that prevents database copies to be mounted on them, and will exclude them from DACP voting. This can be done on the primary and the secondary sites :


o   On the Primary side :

o   If the mailbox servers in the primary are operational and there is a functioning DC in the primary site, use

              Stop-DatabaseAvailabilityGroup -Identity DAG1 -ActiveDirectorySite NYC

o   If the mailbox servers in the primary site are not operational but there is domain controller in the primary site, use this command for each primary MBX servers:

Stop-DatabaseAvailabilityGroup -Identity DAG1 -MailboxServer E14EX3 –ConfigurationOnly

o   If no DC nor mailbox servers are available in the parent side, then make sure that mailbox servers are shutdown always.

o   If the primary mailbox server are online, make sure the cluster service is set to Disabled or do it yourself.

o   On the Secondary side :

o   We need to tell the secondary site which servers are available during the switch over. This can be done by using the Stop-DatabaseAvailabilityGroup command with the ConfigurationOnly.

  • UM Servers:

If any Unified Messaging servers are in use in the failed datacenter, they must be disabled to prevent call routing to the failed datacenter. You can disable a Unified Messaging server by using the Disable-UMServer cmdlet (for example, Disable-UMServer UM01).

 Alternatively, if you are using a Voice over IP (VoIP) gateway, you can also remove the Unified Messaging server entries from the VoIP gateway, or change the DNS records for the failed servers to point to the IP address of the Unified Messaging servers in the second datacenter if your VoIP gateway is configured to route calls using DNS.

3.2 Activating Mailbox Servers

  • When the primary datacenter is down, the mailbox servers in the secondary site, will try to take ownership of the cluster group and will try to bring the primary Witness server online for couple of time before timing out and failing. This is when the cluster as a whole goes down because of majority issues. Database copies on primary datacenter mailbox servers appears as (Service Shutdown), where   database copies on secondary datacenter mailbox servers appear as (Disconnected and Healthy)
  • The Cluster service must be stopped on each DAG member in the primary datacenter. This can be one of two:
    • If the Primary data center is down, then for sure objective completed
    • If the primary mailbox servers are online, make sure cluster service is stopped and the service is marked as disabled
  • Running Restore-DatabaseAvailabilityGroup which will do two things :
    • Evict Stopped DAG members from cluster
    • Create alternative witness share if not created previously on the DAG level

Restore-DatabaseAvailabilityGroup -Identity DAG1 -ActiveDirectorySite LON –      AlternateWitnessServer EXHUB1 -AlternateWitnessDirectory D:\DAG1

You may need to run the command couple of time until the primary mailbox servers are evicted from the cluster.


Note: the restore command can fail, just wait 5 minutes and run it again. Also you can make sure that the command is being executed on the right domain controller by running:

Set-ADServerSettings –PreferredServer <Domain Controller in Failover Datacenter>

  • Always and at any time, if you want to force the cluster model to refresh (i.e if you open the cluster console from the secondary mailbox server, alternative witness share should appear after you entered the Restore-DatabaseAvailabilityGroup command, if it didn’t reflect in the cluster console, just type Set-DatabaseAvailabilityGroup –Identity DAGName)
  • You should make sure the Witness server and directory are up. Never lose them and avoid restarting them. Make sure Exchange Trusted Subsystem is member of the local administrator group on the Witness server and create a firewall rule on the Witness server if necessary to allow all traffic from the mailbox server to the Witness Server.
  • At this moment, the secondary mailbox server(s) will try to assume the ownership of the cluster group and trying to get the secondary DAG IP online and will keep trying to bring the alternative Witness share online.
  • Use Get-DatabaseAvailabilityGroup cmdlet to make sure the Stopped servers are those mailbox servers in the primary site while started servers are those in the secondary site only.
  • If databases in the secondary site don’t mount automatically, remember to remove any activation blocks on the server level (Set-MailboxServer) or on the database level (Suspend Activation).
  • If still databases didn’t mount correctly, use this command:

Move-ActiveMailboxDatabase –Server FQDNofaServerinPrimarySite –ActivateOnServer FQDNofaServerinDRSite

This command contains many Skip switches that can be handy.This is very important step as it is    like taking ownership of those databases. You can also use :

Move-ActiveMailboxDatabase DatabaseName –ActivateOnServer FQDNofaServerinDRSite

  • We need to choose whether to remove the database copies existing in the primary site to allow log truncation or not. If we choose so, reseeding will be necessary once you fail back to the primary data center.

3.3 Clients

  • Outlook Office clients will act as per the following :
    • If the primary CAS servers are online, CAS servers in the primary site will issue a silent redirect message to outlook users. Outlook users will see a message that they need to restart their outlook.
    • If the primary CAS servers are online, you can change the DNS name for the outlook anywhere name or just force autodiscover to work by repairing outlook profile
  • OWA clients will do the following :
    • If the primary CAS servers are online, silent redirection will happen  since both OWA virtual directories has Integrated Authenticated on them
    • If the primary CAS servers are offline, DNS name for OWA primary should point to secondary and that’s it.
  • If you restarted mailbox servers in the secondary site and/or the Witness server, the DAC bit will be sit to 0 and databases will be shown as Dismounted. If you try to mount them , an error that the replication services on the primary mailbox servers are not online. You may find a problem locating the Active manager also especially if you typed: Get-DatabaseAvailabilityGroup –Identity DAGName – Status. The solution will be forcing the DAC bit to be 1 by running the Start-DatabaseAvabilibityGroup –Server (Secondary Mailbox Servers) even if they are already started.


Exchange 2010 DR (Disaster Recovery and Data Center Switch Over) – Unleashed P2

This guide simply explains in a very easy way, all the technologies and procedures that you need to know to perform Exchange 2010 data center switch over, recovering DAG member or stretching DAG between sites.

Check other parts:

Exchange 2010 DR (Disaster Recovery and Data Center Switch Over) – Unleashed P1

Exchange 2010 DR (Disaster Recovery and Data Center Switch Over) – Unleashed P3

Exchange 2010 DR (Disaster Recovery and Data Center Switch Over) – Unleashed P4

Exchange 2010 DR (Disaster Recovery and Data Center Switch Over) – Unleashed P5

2. Datacenter Activation Coordination DAC

2.1 Introduction 

Active Manager handles DAC

DAC mode enables us to use three new commands:  Start-DatabaseAvailabilityGroup, Stop-DatabaseAvailabilityGroup and Restore-DatabaseAvailabilityGroup

DAG uses DACP protocol to handle split brain scenarios when DAG is stretched to more than one subnet.

DAC when enabled, will be an extra application Quorum criteria.

DAC split DAG members to one of two sets:

  1. Stopped DAG Members – Command is  Stop-DatabaseAvailabilityGroup
  2. Started DAG Members  – Command is  Start-DatabaseAvailabilityGroup

Only Started DAG Members will participate in DAC voting. Started servers are those candidate to bring their database copies online.

Stopped DAG member is the status of Active Manager that prevents the databases to be mounted on the server and will exclude it from DAC voting.

Note: So this might seems confusing to most of you. In simple words, when you enable DAC on your DAG, then it is not enough to have normal Cluster Quorum majority to bring databases online. Instead, we have to also test our servers for DAC.

2.2 How to get DAC OK status?

o   If all started DAG members can communicate to each other

o   If not, if a DAG Started member can communicate with a node with DAC bit 1

Note: In case of two DAG started members in the alternate datacenter exist, the boot time of the alternative witness share server can be used. If the witness boot time is before, DAC succeeded. Else, use Restore-DatabaseAvailabilityGroup . This only true for two member started DAG members.

In all cases, if all DAG members are DAC 0, use Start-DatabaseAvailabilityGroup to reset the DAC bit to 1 even if the nodes are already started.

Note: Again let me explain more. Suppose you have 5 DAG servers, SRV1 till SRV5. When you first turn all those servers together, then they will quickly have quorum majority and then will try to check if their DAC test is okay or not. The rule is simple, if all servers can communicate to each others, then each one will stamp itself with DAC value = 1 (Succeed).


Now suppose that SRV1 went down. When you bring it up, it will have DAC =0 and will try to do DAC test: Can SRV1 communicate with at least one server with DAC =1 ? Since SRV2 till SRV5 are all DAC=1, then SRV1 will assign it self with DAC=1 and will mount its databases.




Exchange 2010 DR (Disaster Recovery and Data Center Switch Over) – Unleashed P1

This guide simply explains in a very easy way, all the technologies and procedures that you need to know to perform Exchange 2010 data center switch over, recovering DAG member or stretching DAG between sites.

Check other parts:

Exchange 2010 DR (Disaster Recovery and Data Center Switch Over) – Unleashed P2

Exchange 2010 DR (Disaster Recovery and Data Center Switch Over) – Unleashed P3

Exchange 2010 DR (Disaster Recovery and Data Center Switch Over) – Unleashed P4

Exchange 2010 DR (Disaster Recovery and Data Center Switch Over) – Unleashed P5

1. Introduction Part

1.1 Quorum

Defined as a mechanism to ensure that only one subset of members are functioning at any given time. It is used to find majority.

There is Quorum data that is configuration shared between all nodes.

Exchange 2010 supports only two out of four models of Quorums:

  • Node Majority: for odd number of nodes
  • File share majority: for even number of nodes

Witness is a file share (Witness.log) that represent a vote when there is a need to break the tie. When we are one vote from losing the majority, the node that holds the cluster group (PAM) will lock the witness file share.

The witness cluster file share is created when the DAG members become even and cluster will apply isalive controls to monitor it. If it fails, the cluster group is moved to another node and try to bring it online.

(Exchange Subsystem) group should be member of the local administrator group on the witness server and the alternative witness server.

1.2 DAG Networks

For each subnet that the cluster discovers, a DAG network is created. Note also that heartbeat happens in all networks.

Two types of DAG Networks:

  • MAPI Network:

o   You can have only one MAPI network.

o   Default Gateway and register in DNS

  • Replication Network: (Over TCP 64327)

o   You can have Zero or as many  replication networks as you much

o   No default Gateway and no register in DNS

It is important to note the following:

o   DAG Network enumeration happens only when adding DAG members or can be triggered by running (Set-DatabaseAvailabilityGroup –DiscoverNetworks)

o   If the MAPI network dies in a server, automatic switch over happens.

o   If Replication network dies in a server, replication will happen over MAPI network.

o   ISCIS network should be configured to be ignored from Cluster use.

And also make sure that the replication cannot route to the MAPI network in any case, or cross heartbeat scenario will happen.

1.3 Active Manager

Lives inside (Microsoft Replication Service)

The data about where the database is active now DOES NOT LIVE IN Active Directory. Active Manager is the one who knows about it.

Three Server types:

  1. Standalone ( for nodes not member of DAG)
  2. Standby (SAM)
    1. Monitor local resources and notify PAM
    2. Give information to Active Manager clients about where databases are active
  3. Primary (PAM)
    1. The one who holds the cluster group
    2. Best Copy Selection

Active Manager Client exists in HUB and CAS to know where the active copy lives in order to deliver or access data.