Exchange 2016 Hybrid : TLS negotiation failed with error UnknownCredenta

 

I was adding couple of Exchange 2016 servers with CU2 to the Hybrid configuration wizard to send and receive emails to Exchange Online. On Exchange Online Admin center, I configured the receive connector to Office 365 o verify the subject name on the certificate for TLS authentication.

The problem is that emails are not being sent to Office 365 via the send connector. After enabling the protocol logging on my Exchange 2016 hybrid servers [Get-SendConnector “outbound to Office 365” |Set-SendConnector -ProtocolLoggingLevel verbose] , and opening the smtpsend log file, I can see many TLS failures:

016-07-19T12:13:14.863Z,Outbound to Office 365,08D3AFC581A92DD3,3,10.28.2.202:8105,213.199.154.87:25,>,EHLO mail.contoso.com,
2016-07-19T12:13:14.910Z,Outbound to Office 365,08D3AFC581A92DD4,2,10.28.2.202:8106,213.199.154.87:25,<,”220 DB3FFO11FD036.mail.protection.outlook.com Microsoft ESMTP MAIL Service ready at Tue, 19 Jul 2016 12:13:14 +0000″,
2016-07-19T12:13:14.910Z,Outbound to Office 365,08D3AFC581A92DD4,3,10.28.2.202:8106,213.199.154.87:25,>,EHLO mail.contoso.com,
2016-07-19T12:13:15.004Z,Outbound to Office 365,08D3AFC581A92DD3,4,10.28.2.202:8105,213.199.154.87:25,<,250 DB3FFO11FD029.mail.protection.outlook.com Hello [86.96.206.50] SIZE 157286400 PIPELINING DSN ENHANCEDSTATUSCODES STARTTLS 8BITMIME BINARYMIME CHUNKING,
2016-07-19T12:13:15.004Z,Outbound to Office 365,08D3AFC581A92DD3,5,10.28.2.202:8105,213.199.154.87:25,>,STARTTLS,
2016-07-19T12:13:15.051Z,Outbound to Office 365,08D3AFC581A92DD4,4,10.28.2.202:8106,213.199.154.87:25,<,250 DB3FFO11FD036.mail.protection.outlook.com Hello [86.96.206.50] SIZE 157286400 PIPELINING DSN ENHANCEDSTATUSCODES STARTTLS 8BITMIME BINARYMIME CHUNKING,
2016-07-19T12:13:15.051Z,Outbound to Office 365,08D3AFC581A92DD4,5,10.28.2.202:8106,213.199.154.87:25,>,STARTTLS,
2016-07-19T12:13:15.145Z,Outbound to Office 365,08D3AFC581A92DD3,6,10.28.2.202:8105,213.199.154.87:25,<,220 2.0.0 SMTP server ready,
2016-07-19T12:13:15.145Z,Outbound to Office 365,08D3AFC581A92DD3,7,10.28.2.202:8105,213.199.154.87:25,*,” CN=*.contoso.com, OU=IT, O=contoso International (L.L.C), L=Dubai, S=Dubai, C=AE CN=thawte SHA256 SSL CA, O=””thawte, Inc.””, C=US 0D92CFF6070B73AD5722EC8B4DA3389B AAA3D3DADA6891A2CCB3134D0B2D7764F1351BC4 *.contoso.com”,Sending certificate Certificate subject Certificate issuer name Certificate serial number Certificate thumbprint Certificate subject alternate names
2016-07-19T12:13:15.145Z,Outbound to Office 365,08D3AFC581A92DD3,8,10.28.2.202:8105,213.199.154.87:25,*,,TLS negotiation failed with error UnknownCredentials

I am sure the certificate is fine as the other hybrid servers are using the same certificate and they are able to send emails to Office 365. Also on the event viewer, I am seeing the following error:

TLS Error Office 365 Exchange Hybrid

 

So finally, I tried something and it worked. I opened the certificate store, and I was checking the permissions on my certificate private key, the certificate I am using for the TLS connection.

TLS Error Office 365 Exchange Hybrid2

I can see the following permissions on the private key:

TLS Error Office 365 Exchange Hybrid3

 

So I added the Network Service and I gave it READ access. After that everything worked just fine. Try to give EVERYONE Read access if things are not working yet.

Hope this will help someone, leave a note if it did ūüôā

Office 365 and Group Moderation Tips

In the process of testing out Office 365 and Exchange hybrid configuration, an interesting thing happened that I want to share with you.

I have an on-premise Exchange 2010 implementation and couple of users are hosted at Office 365. All hybrid configurations are set and connectors are configured to route emails between the two spaces.

Everything is working fine, and mailboxes hosted on Office 365 are working just fine. Things started to get interesting when people start to send emails to moderated distribution groups.

When someone sends email to a moderated groups, and the moderator is hosted on Office 365, the buttons for Approve and Reject are not showing at his email client.

It turned out that a setting called TNEF (Transport Neutral Encapsulation Format) is causing this to happen. We need to make sure TNEF format is enabled when sending emails out to Office 365 tenant.

The TNEF setting is configurable per remote domain (Get-RemoteDomain) and (Set-RemoteDomain).

By default, there is a default RemoteDomain configured in your Exchange environment called (Default). If you hit (Get-RemoteDomain), you will see all settings that controls the behavior of email communications and format when sending emails to external parties. One of the settings is TNEFEnabled.

Now that we have Office 365 hybrid setup, the HCW creates for us a remote domain in the on-premise organization to allow TNEF (TenantName.mail.onmicrosoft.com).

That is great. So all what we need to do is to configure that remote domain (Set-RemoteDomain -TNEFEnabled ….) and all is done, right?

There is a small thing left to say. When Office 365 sends emails regarding moderated groups, the messages come from a system mailbox in the tenant with email address SystemMailbox{bb558c35-97f1-4cb9-8ff7-d53741dc928c}@TenantName.onmicrosoft.com. So we need to add a new remote domain called TenantName.onmicrosoft.com

So let us start typing some PowerShell commands:

New-RemoteDomain -Name “Hybrid Domain –TenantName.onmicrosoft.com” -DomainName TenantName.onmicrosoft.com

Now we have two remote domains:

  • TenantName.onmicrosoft.com
  • TenantName.mail.onmicrosoft.com

We have then to configure both remote domains to allow TNEF format. I also recommend configuring many other settings on the way.

Set-RemoteDomain -Name “Hybrid*” -IsInternal $true -TargetDeliveryDomain $true -AllowedOOFType InternalLegacy -MeetingForwardNotificationEnabled $true -TrustedMailOutboundEnabled $true -TrustedMailInboundEnabled $true -UseSimpleDisplayName $true -TNEFEnabled $true

That’s great. Now we have configured both remote domains to enable TNEF format. I have also noticed that when on premise mailboxes try to communicate with the Office 365 system mailbox for moderation actions (Approve,Reject), they are receiving authentication errors. To fix that, add the TenantName.onmicrosoft.com to the address space of the Office 365 connector:

Set-SendConnector “Outbound to Office 365″ -AddressSpaces @{Add=”TenantName.onmicrosoft.com”}

Finally, it makes sense to instruct the Office 365 tenant to treat the on premise Exchange organization the same way. Suppose my on premise domain domain is contoso.com, then connect to your office 365 Exchange PowerShell, and type:

New-RemoteDomain -Name “Hybrid Domain – Contoso.com” -DomainName Contoso.com

Set-RemoteDomain -Name “Hybrid*” -IsInternal $true -TargetDeliveryDomain $true -AllowedOOFType InternalLegacy -MeetingForwardNotificationEnabled $true -TrustedMailOutboundEnabled $true -TrustedMailInboundEnabled $true -UseSimpleDisplayName $true -TNEFEnabled $true.

Reference Link 

Azure Sync Case – Conflict between UPN and SMTP address

I was working heavily on AD Sync Tool with all its versions including the AAD Connect tool. And I came across an issue that will cause you trouble soon. So in this blog post I will share my experience with you.

Case:

My environment was consisting of one AD domain with Exchange 2010 on premise and a single sign on experience. The AD domain and SMTP domain is the same (contoso.com)

I have a manager called John Smith:

  • UPN :¬†JohnS@contoso.com
  • SMTP address : John.Smith@contoso.com

This manager came to me asking me to add a secondary SMTP address for him John@contoso.com. Since that SMTP address is available, I agreed.

Azure AAD Sync UPN vs SMTP 12121

Life goes on. John is a big manager and he used his nice clean new SMTP John@contoso.com in all his communications. He printed the new email address to his business cards.

So far, we have all our mailboxes are on premise and we do not have any Office 365 implementation.

The Problem

After 10 years, the corporate decided to start using Azure services and they decided to start with AD Sync next month.

Meanwhile, a new employee is hired with name John William. The IT department assigned him the following:

  • UPN : John@Contoso.com
  • SamAccountName : Contoso\John
  • SMTP Address : John.William@Contoso.com

Everything is fine so far and there is no single conflict.

Azure AAD Sync UPN vs SMTP 1212122

Now when we started to Sync users to Azure AAD, John Smith user is no longer synchronizing to Azure AD and the AD sync tool is giving an error for that user.

“Unable to update this object because the following attributes associated with this object have values that may already be associated with another object in your local directory services:[ProxyAddresses smtp: JohnS@contoso.com].Correct or remove the duplicate values in your local directory.”

After opening a case with Microsoft, we reviewed our AD Sync configuration and after opening the AAD Connect sync tool, we confirmed that we are using ObjectGUID as the anchor attribute and not email address.

Microsoft confirms that when an on premise mail enabled user is synched to Azure AD, he is assigned a secondary SMTP in the form of his UPN. So in this case, when John William is being synched to Azure AD, Azure will try to stamp him with a secondary SMTP in the form of John@Contoso.com which causes a conflict with John Smith user.

So although there is no conflict on premise, Azure introduces this type of conflict and throwing sync error. ¬†Microsoft answered us : “THIS IS BY DESIGN DEAL WITH IT”

How big this problem is

Now, we contacted John Smith and asked him to give away his lovely John@Contoso.com SMTP address. What do you think his reaction is ? He is using that email address since years and he cannot give it away. Further more, if he agreed to give it away, and if Azure assigned it to John William user, then people sending emails to John@contoso.com will be sending to John William and this is by itself a security and privacy issue.

So we went to John William and ask him to change his UPN and SamAccountName. This solves the problem for a while, until a new employee come again with the name John Robert, and the IT found that Contoso\John is not used in the enterprise and so they assigned it to John Robert.

Now suddenly the Sync Tool will throw a sync error and the same loop happens over and over.

Imagine you have many users who used to have clean SMTP addresses and you have to go through them one by one and do this ūüôā

Compare your on Premise AD users and Azure users – One by One !

If you are synchronizing your on premise Active Directory with Azure Active Directory, then you know for sure that maintaining a healthy synchronization is not an easy thing to do.

I want to give you my experience in synchronizing a single domain AD to Azure AD, using ObjectGuid as anchor attribute.

The Challenge

I am using Azure AD Connect to synchronize AD objects to Azure AD. The Azure AD Connect tool is nice, and it gives you the health of the synchronization process and a nice error messages if there is a problem synchronizing one of your on premise AD objects. I used the AD connect tool for a month, and everything was fine. No errors and everything looks clean.

I then noticed that couple of users are having problem with Office 365 and i discovered that they have never synched to Azure AD in the first place. I did Get-MSOLUser on them and no results are returned. I had to go to Azure AD Connect and force Full AD Import to sort out this issue.

I become so worried about this, and I started to compare the number of AD users and Azure AD users to at least ensure matching numbers. Count AD users and Azure AD users and compare them is not enough, as there is no guarantee that the same objects represent these numbers.

I then came with an idea. I wanted to take each AD user, collect his ObjectGuid, and then compute his ImmutableID, go to Azure AD, and search for that ImmutableID, and finally linked the AD user with his Azure AD copy. We will do that for all AD users. Finally, we can identify AD users that does not have Azure copies, and Azure AD users that are not mapped to AD users.

This will guarantee that each one of your AD users are mapped to Azure AD users. Along this journey, the script will generate couple of information:

  • AD users not in Azure
  • Azure users not in AD
  • Azure users with Synch Errors
  • Filtered AD users from Synchronization
  • Total number of AD users
  • Total number of Azure users
  • Last AD Sync time

Compare you on Premise AD users and Azure users 11

Conditions to use the script

  1. This script assumes you are using ObjectGuid as anchor attribute
  2. If you have multiple domains, forests or you are doing filters to scope AD users by OU or attribute, then you must write your script block to populate the $ADusers_Raw variable, by searching inside the script for (Raw data collection from AD) region. You can find examples there to do that. By default the script will do Get-ADuser to retrieve all users.

Compare you on Premise AD users and Azure users 2

Script Visuals

The script provides many visuals to help you see what is going on:

  • The script code is divided into regions that you can expand separately for better script browsing.
  • Progress bars will appear during running the script to give you a feel and sense of what is going on during the run time of the script.
  • Summary information is displayed on the PowerShell console window with summary statistics.
  • Results are written at the end of the script in two locations:
    • The PowerShell console window.
    • Couple of files that are generated on the script running directory.

Notes

The script connects to Active Directory and gets a list of users (Get-ADUser) , and then connects to Azure Active Directory AAD and gets all azure users (Get-MSOLUser). A comparison then is performed by mapping each AD user with his Azure user, and identify un-synchronized AD users that do not have a mapped/synched Azure copy.

This script assumes that ObjectGUID is used as the anchor attribute to link/map AD user and Azure AAD user. If you are using any other attribute, then this script is not for your case.

The script needs to get the list of AD users that you are synchronizing to Azure AD. Microsoft Sync tool gives you the ability to filter by OU or by attribute. Another tricky part of the script is when you are synchronizing multiple domains or perhaps multiple forests. For all those different cases, it is your job to populate the variable called ($ADusers_Raw) located under (Raw data collection from AD) region in this script.
By default, the script will do Get-ADUser to populate this variable. In your case, you may need to write your own script block to collect all AD user objects that you are synching to azure.

Script Parameters

.PARAMETER ScriptFilesPath
Path to store the script output files. For example C:\ , or ‘.\’ to represent the current directory.

.PARAMETER Proxy
As the script needs to connect to Azure, an internet connectivity is required. Use this option if you have an internet proxy that prompts for credentials.

.PARAMETER CreateGlobalObject
The switch when used, the script will generate an extra csv file that contains a unified view of each user with properties from the on premise AD and Azure AD.

Examples

.EXAMPLE
.\Compare-CorpAzureIDs.ps1 -ScriptFilesPath .\

.EXAMPLE
.\Compare-CorpAzureIDs.ps1 -ScriptFilesPath .\ -Proxy:$true

.EXAMPLE
.\Compare-CorpAzureIDs.ps1 -ScriptFilesPath .\ -CreateGlobalObject

Download the script

You can download the script from here.

Office 365 – This may indicate invalid parameters in your hybrid configuration settings.

I came across a case in which an organization wants to implement Office 365 Hybrid Exchange, and they have a tenant with the following licenses (subscriptions): (EOP,EOA,DLP).

They initiated the Exchange Hybrid wizard and the wizard reported the following error:

INFO : Session=Tenant Cmdlet=Set-OnPremisesOrganization -HybridDomains {contoso.com} -InboundConnector 'Inbound from 866c6111-4067-46e1-bb8b-df0637594a40' -OutboundConnector 'Outbound to 866c6111-4067-46e1-bb8b-df0637594a40' -OrganizationRelationship 'O365 to On-premises - 866c6111-4067-46e1-bb8b-df0637594a40' -OrganizationName 'contoso' -Identity '866c6f7c-4067-46e1-bb8b-df0637594a40' START
INFO : Session=Tenant Cmdlet=Set-OnPremisesOrganization FINISH Time=781.2552ms
ERROR : Subtask Configure execution failed: Configure Mail Flow
Execution of the Set-OnPremisesOrganization cmdlet has thrown an exception. This may indicate invalid parameters in your hybrid configuration settings.
A parameter cannot be found that matches parameter name 'HybridDomains'.
at System.Management.Automation.PowerShell.CoreInvokeRemoteHelper[TInput,TOutput](PSDataCollection`1 input, PSDataCollection`1 output, PSInvocationSettings settings)
at System.Management.Automation.PowerShell.CoreInvoke[TInput,TOutput](PSDataCollection`1 input, PSDataCollection`1 output, PSInvocationSettings settings)
at System.Management.Automation.PowerShell.CoreInvoke[TOutput](IEnumerable input, PSDataCollection`1 output, PSInvocationSettings settings)
at System.Management.Automation.PowerShell.Invoke(IEnumerable input, PSInvocationSettings settings)
at Microsoft.Exchange.Management.Hybrid.RemotePowershellSession.RunCommand(String cmdlet, SessionParameters parameters, Boolean ignoreNotFoundErrors)

 

We have verified that the user running the Hybrid wizard is Global Administrator on the tenant, and the strange thing is that the error specified that the Set–OnPremisesOrganization command does not have a parameter called HypridDomains. If you check TechNet, you can see this is not true. So how come?

Solution

After spending could of days with Microsoft support,  it seems that we have to add at least one E3 Office 365 licenses to the tenant in order for the PowerShell commands to work properly.