This article is all about talking about the email moderation when people have mailboxes on premise and on Office 365. In certain cases, the approval process do not work well if we have mixed of approval and senders between on premise and Office 365.
How Email Moderation Works
- A mail user sends an email to a moderated group.
- The categorizer at hub transport server intercepts the email, marks for moderation and then re-routes it to the arbitration mailbox.
- The store driver component stores the message in the arbitration mailbox and sends an approval request to the moderator.
- The moderator takes an action.
- The store driver marks the moderator’s decision on the original message stored in the arbitration mailbox.
- The Information Assistant reads the approval status on the message stored in the arbitration mailbox, and then process the message depending on the moderator’s decision.
- If the moderator has approved the message, the Information Assistant resubmits the message to the submission queue, and the message is delivered to the recipient(s).
- If the moderator has rejected the message, the Information Assistant deletes the message from the arbitration mailbox and notifies the sender that the message was rejected.
So What are the Arbitration Mailboxes?
Those are system mailboxes that get created by default in any Exchange organization. They have a well known names that is similar in any Exchange organization.
You can view them by typing Get-Mailbox -Arbitration
The one that we are interesting in here is the first one (Microsoft Exchange Approval Assistant). This is the system mailbox that handles moderation.
IT Admin can create as many arbitration mailboxes for moderation to load balance the approval process, but we choose to stick with this one default one.
Notice that on our on premise Exchange implementation, we know we have only one arbitration mailbox for moderation and we know its address by typing Get-Mailbox – Arbitration.
In Exchange Online, the property -Arbitration does not exist in the Get-Mailbox command. This is because it is hosted environment and Microsoft seems to have lots of those arbitration mailboxes and they are hidden from us. We just know that some arbitration mailbox in the cloud will handle the approval process, we just know which one and we cannot query it.
IMPORTANT NOTE: We only know and this is so important, that the Exchange Online arbitration mailbox has email address that ends with @contoso.onmicrosoft.com and NOT @contoso.mail.onmicrosoft.com. Strange enough as we all know that contoso.onmirosoft.com is identity domain, while @contoso.mail.onmicrosoft.com is a mail routing domain.
The One Million Dollars Question: Which Arbitration Mailbox will be used?
So we have people in Office 365 and their mailboxes are in Exchange Online, and we have people on premise with mailboxes in our on premise Exchange Environment, so which arbitration mailbox will kick in when sending to a moderated group? The Exchange Online one or the one we know about in our Exchange on premise system?
Why this is important is coming soon, but it is so important to understand which arbitration mailbox will be invoked.
There are two types of moderated groups on our on premise Exchange system:
- Dynamic groups that are moderated.
- Non-Dynamic groups that are moderated.
Why this is important? Well, because the way AD Sync works.
Dynamic groups created on premise are not synchronized to Azure AD. Azure AD does not recognized our dynamic groups. This means that all our dynamic groups are not available in the address book of Office 365 mailbox users.
Well, we kind of trick Office 365 mailbox users by creating contacts on Azure AD that has the same name of each dynamic group that we have on premise so that they can see ALL GSO for example in their address book.
So why this is important from our subject perspective. Well, the only thing that we want to make sure we understand is that when Office 365 hosted mailbox user is sending email to an on premise dynamic group, there is no way for Exchange Online to know that this group is moderated because AD Sync does not sync dynamic groups to Azure AD, instead it will route it to our Exchange on premise servers to deal with it. Now our Exchange on premise knows that this is a moderated group, and it will put that email to the on premise arbitration mailbox. Sound good?!
While if a an Office 365 hosted mailbox user is sending email to an on premise non-dynamic group, Exchange Online query Azure AD, and since this group is synced, Exchange Online at this moment knows and realize this is a moderated group, because the group and all of its properties including moderation information is synced and available in Azure AD. So Exchange Online will try to behave in a smart way and start taking responsibility. It will not dare to forward the email to Exchange on premise, as it feels that it should stand to the challenge. It knows it is moderated group, so it will try to save the day. It will shout (Hey Exchange Online, give me an available arbitration mailbox, I have work to be done), and it will through the email to one of the many available arbitration mailboxes in Exchange Online, that we do not know their names or specific email address. We only dare to know that the email address of those cloud arbitration mailboxes ends with @contoso.onmicrosoft.com (yes not @mail.contoso.onmicrosoft.com)
Before anything start working, there are couple of prerequisites to be accomplished before making moderation works between on premise and Exchange Online.
- On the outbound connector to Office 365, we shall add @contoso.onmicrosoft.com in the address space for that connector. That is we need to educate our Exchange on premise that anything going to @contoso.onmicrosoft.com should be routed to Exchange Online. This is necessary for the approval messages from on premise moderators to reach the Exchange Online arbitration mailbox (that has email address ends with @contoso.onmicrosoft.com).
- In order for the button called Approve and Reject to work correctly in hybrid moderation, we need to configure remote domains and configure them in both on premise and Exchange Online. This would be mentioned in part 2 of this article.
How things works?
Well, we have 8 possibility to take care of when evaluating how hybrid moderation works.
- On Premise sending to On Premise Dynamic Group and the approval is On Premise. [We know it will work]
- On Premise sending to On Premise Dynamic Group and the approval is On Cloud.
- On Cloud sending to On Premise Dynamic Group and the approval is On Premise.
- On Cloud sending to On Premise Dynamic Group and the approval is On Cloud.
- On Premise sending to On Premise Non-Dynamic Group and the approval is On Premise.[We know it will work]
- On Premise sending to On Premise Non-Dynamic Group and the approval is On Cloud.
- On Cloud sending to On Premise Non-Dynamic Group and the approval is On Premise.
- On Cloud sending to On Premise Non-Dynamic Group and the approval is On Cloud.
We will go through those cases and try to see which arbitration mailbox will be invoked and what to expect. The rule is [THE FIRST EXCHANGE SERVER WHO KNOWS ABOUT MODERATION, WILL USE HIS ARBITRATION MAILBOX]
Case 2: On Premise sending to On Premise Dynamic Group and the approval is On Cloud.
Since the sender [Let us say Alice] is on premise, and the on premise Exchange knows about the fact that the dynamic group is moderated, it will use the on premise arbitration mailbox. So the email will be forwarded to the on premise arbitration mailbox.
The arbitration mailbox will send email to the moderator [say his name is Bob and his mailbox is in Office 365]. Bob will receive email from [MSExchApproval1f05a927email@example.com] saying hey please approve this email.
Bob when he approves the email, he is like sending email to that arbitration mailbox, so it is Bob’s mailbox sending email to MSExchApproval1f05a927firstname.lastname@example.org. All sounds good. Arbitration mailbox will release the email, letting the on premise Exchange transport engine to deliver the email to that group.
Case 3: On Cloud sending to On Premise Dynamic Group and the approval is On Premise.
Now this is interesting. Alice is Office 365 user and trying to send email to dynamic group on premise that is moderated. Since Exchange Online do not know anything about Dynamic Groups created on premise because they are not synced to Azure AD, it just forward the email as is to on premise Exchange.
On premise Exchange will send the email to the on premise arbitration mailbox, approval will be sent to the on premise moderator from the on premise arbitration mailbox and everything works fine here. The on premise moderator will approve the email, so he will be replying to the on premise arbitration mailbox, and then the transport engine will release the message.
Case 4: On Cloud sending to On Premise Dynamic Group and the approval is On Cloud.
This is similar to the previous case. the on premises arbitration mailbox will be used as the Exchange Online system does not know that the group is moderated as it is not synced to Azure AD. Of course now the on premise arbitration mailbox will send approval email to the cloud moderator. The cloud moderator will see the email coming from the on premise moderation mailbox, he will reply back by approving or rejecting, and all will work just fine.
Case 6: On Premise sending to On Premise Non-Dynamic Group and the approval is On Cloud.
Alice is on premise user, trying to send email to a non-dynamic group on premise, so the on premise Exchange will pickup the message, and since it is aware that the group is moderated, it will send the message to the on premise arbitration mailbox. Nothing interesting here and everything will work just fine.
Case 7: On Cloud sending to On Premise Non-Dynamic Group and the approval is On Premise.
THIS IS SO IMPORTANT CASE: IT WILL BREAK AND NOT WORK !!!
So now Alice is Office 365 user trying to send email to on premise non-dynamic moderated group. Since Alice is in Office 365, Exchange Online is the first point of contact here, and it knows, yes it knows, that that on premise group is moderated. Why? because it is synced to Azure AD, in contrast to the dynamic groups which are not synced.
So here we go!. Exchange Online will send the email to the Exchange Online arbitration mailbox. Remember the rule: [THE FIRST EXCHANGE SERVER WHO KNOWS ABOUT MODERATION, WILL USE HIS ARBITRATION MAILBOX].
We only know that the Exchange Online arbitration mailbox has an email address that ends with @contoso.onmicrosoft.com.
Now the arbitration mailbox in Office 365 will send email to Bob the moderator whose mailbox is located on the on premise Exchange system.
The email looks like this:
Please approve or reject this message from Alice to this group.
Bob is a good dude, we will click APPROVE. and let us pause for a second here. Bob by pressing APPROVE, he is like sending email that looks like this:
Please approve or reject this message from Alice to this group.
Guess who will be sending this gameness? Of course Exchange on premise, and since there is a connector between us and Exchange Online with @contoso.onmicrosoft.com as a destination SMTP address space, out hybrid servers will forward the message to Exchange Online.
Guess who is protecting Exchange Online? Yes it is Exchange Online Protection EOP. This EOP system has a wall of defense called Accepted Domains and Directory Based Edge Blocking. Also the domain contoso.onmicrosoft.com is defined as Authoritative domain in Exchange Online.
So what all that means? Well, in Exchange Online, Accepted Domains are the SMTP domains that EOP receives email to. When a domain is configured as Authoritative Accepted Domain, EOP applies a firewall defense called Dictionary Based Edge Blocking, that does the following:
- If I receive an email coming to an authoritative domain, the recipient in that email should be well defined as a recipient in the Azure AD [Exchange Online Address Book].
- Else, drop the email.
So what is the point here? This is to protect from dictionary attack, where someone tries to open a dictionary and tries every possible first and last name, and tries to send storm of possible smtp addresses for a domain. If this dictionary based edge blocking is not in place, then EOP would send all those attempts to the internal Exchange system to deal with, and filter wrong address from right ones, which consume the resources of the internal email system.
To return back and talk about our case, now the email is coming from on premise Exchange from Bob@contoso.com and going to some Guid address that ends with @contoso.onmicrosoft.com via the outbound connector to Office 365. Now EOP will check the domain @contoso.onmicrosoft.com and it will see it configured as Authoritative domain, and will apply the Dictionary Based Edge Blocking knowledge. So it will look at the Azure AD and will ask Azure AD ” Hey Azure AD, do you have a recipient with email address StrangeGuid@contoso.onmicrosoft.com”. Azure AD will say “hell, no”, and EOP will block that message. Interesting right? This took us long time to figure it out.
Case 8: On Cloud sending to On Premise Non-Dynamic Group and the approval is On Cloud.
This is straight forward, so the Exchange Online arbitration will pickup the message, and communicate with the cloud moderator, and all will work just fine. No traffic will reach the on premise Exchange environment until the final group delivery in case the group contains on premise recipients.
Why don’t turn Dictionary Based Edge Blocking off for email@example.com?
We can do that by going to Exchange Online> Accepted Domains, and then we can make the @contoso.onmicrosoft.com as Internal Relay domain and not Authoritative. This will turn off the Dictionary Based Edge Blocking. Going back to case 7 above, when EOP receives a message going to StrangeGuid@contoso.onmicrosoft.com, it will not bother asking Azure AD anything, and will just deliver the message to Exchange Online. Exchange Online knows this address, as it is one of many arbitration mailboxes in Office 365, and everything will work just fine.
Wake Up !! This should never ever been done. the domain @contoso.onmicrosoft.com should always be authoritative domain or other stuff will stop working.
The first thing that will not work the moment you switch that domain to Internal Relay is the Hosted Voice Mail. Hosted Voice Mail REQUIRES that the domain @contoso.onmicrosoft.com to be Authoritative. There is no clear reason why this is required, but it took us long troubleshooting time to figure this out and fix the hosted voice mail system.
Note: Hosted Voice Mail means that we have people with Office 365 mailboxes and using the on premise Skype for Business telephony, and we need the Skype for Business on premise to deliver voice mail for Office 365 through Skype federation with Exchange Online. This requires the domain @contoso.onmirosoft.com to be configured as Authoritative domain in Exchange Online Admin Console.
Other Solution, creating contacts
So the other option that some Microsoft support engineer proposed an then we figured out it would not make sense is to create a contact in our Exchange Online system, mail enabled contact with an email address StrangeGuid@contoso.onmicrosoft.com.
StrangeGuid@contoso.onmicrosoft.com is the smtp address of the Exchange Online arbitration mailbox.
Now going back to case 7 above, when Bob sends email to StrangeGuid@contoso.onmicrosoft.com and since that domain is authoritative and the Dictionary Based Edge Blocking logic will kick, EOP will try to match that recipient address with any recipient defined in Azure AD, and it will found that mail enabled contact that we just made, and it will deliver the mail to Exchange Online that knows how to deal with it.
This will work will except that there are many arbitration mailboxes in Exchange Online and we do not know how to query them. The best guess is to try to simulate the Case 7 over and over and for each new arbitration mailbox Online that we detect, we shall create a mail enabled contact in Exchange Online to trick EOP. This does not work well in the long run.
What to do?
The best thing to do is to try to scope the problem. We know we have 7 out of 8 cases working just fine, and we have only on case not working fine. Only when the moderator is on premise we have an issue.
We also do not have any issues with on premise dynamic groups, only non-dynamic groups with on premise moderator is where we need to look at.
One solution could be migrate the mailbox of all on premise moderators for non dynamic groups. This can play well, but needs some time which we cannot have now.
The other solution is this. Remember why we do not have problem with on premise dynamic groups even though they have on premise moderators? Well, because Exchange Online do not know about them, because they are not synced to Azure AD in the first place, which in place triggers the email to go to the on premise arbitration mailbox, which works just fine.
The proposed solution is to query our directory for all non-dynamic on premise moderated groups, that have on premise moderators, and put them in an OU that is not synced to Azure AD, and then create Exchange contacts for them on Exchange Online. This will solve the whole issue for now until all moderators are in Office 365.
The only bad thing about this is that Office 365 users will not be able to expand those groups because they appear as contacts, and will not see the mail tip that the group is moderated (which we can mimic by using mail tips). This will work well I believe for the time being.
Now, when Office 365 users trying to send to one of those groups, they will open address book, and see a group called firstname.lastname@example.org. The fact is this is a mail enabled contact, so Exchange Online will not know it is moderated, and will just send it to Exchange on premise, which will send it to the on premise arbitration mailbox and everything will work just fine.