HCW v3 Stuck at "Adding Federated Domain"

Exchange Hybrid Configuration Wizard - always interesting times every single time I run this wizard.

My customer has three public facing domains that is required as part of the Exchange Online migration. I've added them into Verified Domain in Office 365 portal, ran HCW v3, added the TXT records required for Microsoft Federation Gateway, and when I clicked on "Verify TXT record" button, all 3 domain changed into "TXT record found", and the first domain proceed to be stuck at "Adding federated domain".

Looking into the HCW log (always a good place to start with %appdata%\Roaming\Microsoft\Exchange Hybrid Configuration\), the following error was found:

2017.04.20 01:08:12.023 *ERROR* [Client=UX, Page=DomainProof, Thread=17] Microsoft.Online.CSE.Hybrid.Provider.PowerShell.PowerShellInvokeException: PowerShell failed to invoke 'Set-FederatedOrganizationIdentifier': Unable to reserve domain "FYDIBOHF25SPDLT.company.com" for Application Identifier "000000004C04A704".  Detailed information: "A Windows Live ID error occurred. Detailed information: "PassportError: Passport error.".". ---> System.Management.Automation.RemoteException: Unable to reserve domain "FYDIBOHF25SPDLT.company.com" for Application Identifier "000000004C04A704".  Detailed information: "A Windows Live ID error occurred. Detailed information: "PassportError: Passport error.".".

                                   --- End of inner exception stack trace ---

                                   at Microsoft.Online.CSE.Hybrid.PowerShell.RemotePowershellSession.RunCommandInternal(Cmdlet cmdlet, SessionParameters parameters, Int32 millisecondsTimeout, PowerShellRetrySettings retrySettings, Boolean skipCmdletLogging)

                                   at Microsoft.Online.CSE.Hybrid.Session.PowerShellOnPremisesSession.SetFederatedOrganizationIdentifier(SmtpDomain accountNamespace, String delegationTrustLink, SmtpDomain defaultDomain)

                                   at Microsoft.Online.CSE.Hybrid.App.ViewModel.Pages.DomainProof.DomainInfo.AddFederatedDomain(IOnPremisesSession session, AppData appData)

                                   at System.Collections.Generic.List`1.ForEach(Action`1 action)

                                   at Microsoft.Online.CSE.Hybrid.App.ViewModel.Pages.DomainProof.VerifyActivity(IOnPremisesSession session, IEnvironment environment)

Restarting HCW did not help.

Resolution: Verify one domain at a time. In the page "Select the domains that you want to be part of your Hybrid Configuration", pick only one domain, and click Next, and proceed to verify that domain. Using the Back button, go back and pick another one (and unselecting the previous one) and verify that domain, and repeat for the last domain.

Hope this helps someone.

OWA Redirection Does Not Work for Migrated Mailbox to Exchange Online

So you have migrated a user to Exchange Online (via hybrid configuration). You told the user upfront that they should still be able to use the existing OWA hyperlink and will get a page reminding them that their mailbox has migrated to Exchange Online and bookmark outlook.office365.com instead.

But instead, the user is presented with this page with a sad face instead:

The error message is:

We could not find a mailbox for this user. Either this recipient has not been configured with a user mailbox or does not have a license assigned. Please contact your helpdesk for further assistance.

X-OWA-Error: Microsoft.Exchange.Clients.Owa2.Server.Core.OwaUserHasNoMailboxAndNotLicenseAssignedException

Of course you have assigned Exchange Online licenses (or E3), and the mailbox has obviously moved to Exchange Online.

The resolution is rather simple but rarely documented.

Clear your browser cache, close the browser and login again.

Boom! The error goes away and the correct page showing the correct URL to be bookmarked.

Exchange Online In-Place Archive not showing up in Outlook Web App

Ever had the problem when you have enabled the user's mailbox with In-Place Archive (a.k.a. Hosted Archive or Archive Mailbox), but yet it doesn't show up in OWA?

And you have tried to enable it for a hybrid mailbox, as well as a cloud only mailbox and both has the same behaviour?

The fix for me was a simple one - just clear your browser cache - for some reason that solved the problem for me.

Exchange Hybrid Free/Busy - Fails after running HCW to add more domains

So you have decided to run HCW again to include additional domains that you have skipped in the initial setup.

And immediately, you notice that free/busy query from Exchange Online to On Premises have started to fail again.

If you have already fixed it from my previous post, why is this happening again?

Firstly, run the following command from Exchange Online PowerShell:

Test-OrganizationRelationship -UserIdentity onPremiseMailbox@company.com -Identity “O365 to On-premises 6633cadc-0124-4111-2a22-e51f8853d1c5” -Verbose

Note that it will fail at STEP 4:

STEP 4: Getting organization relationship settings from remote partner...

RESULT: Unable to retrieve organization relationships from remote organization.
RESULT: Error.

But if you look back at STEP 3 - you will notice that the target URL is probably showing the new domain that you just added:

STEP 3: Requesting delegation token from the STS...

RESULT: Success.

Retrieved token for target https://autodiscover.newcompany.com/autodiscover/autodiscover.svc/WSSecurity for offer Name=MSExchange

.Autodiscover,Duration=28800(secs)

So what's the problem here? Most likely, this is used as a secondary email address and you haven't bothered to configure autodiscover for it.

To confirm this, run the following command from Exchange Online PowerShell:

Get-OrganizationRelationship | FL

Check out the "TargetAutodiscoverEpr" field, it is probably pointing to https://autodiscover.newcompany.com/autodiscover/autodiscover.svc/WSSecurity, instead of https://autodiscover.company.com/autodiscover/autodiscover.svc/WSSecurity

To solve the problem, either configure autodiscover for that domain (add it in public DNS, and update your TMG rules + add the SAN into your certificate), or just repoint it back to the correct autodiscover URL.

This can be done by executing the following command from Exchange Online PowerShell:

Get-OrganizationRelationship | Set-OrganizationRelationship -TargetAutodiscoverEpr https://autodiscover.company.com/autodiscover/autodiscover.svc/WSSecurity

Exchange Online - Mailbox Move Back On Premises Error

So you are trying to move a mailbox from Exchange Online back to your on premises Exchange Server and received the following error:

Error: MigrationTransientException: The call to ‎'https://hybridserver.company.com/EWS/mrsproxy.svc wstcorpcas01.wilson.com ‎(14.3.227.0 caps:05FFFF)‎‎' failed. Error details: The server was unable to process the request due to an internal error. For more information about the error, either turn on IncludeExceptionDetailInFaults ‎(either from ServiceBehaviorAttribute or from the configuration behavior)‎ on the server in order to send the exception information back to the client, or turn on tracing as per the Microsoft .NET Framework 3.0 SDK documentation and inspect the server trace logs.. --> The call to ‎'https://remotewest.wilsongroupau.com/EWS/mrsproxy.svc wstcorpcas01.wilson.com ‎(14.3.227.0 caps:05FFFF)‎‎' failed. Error details: The server was unable to process the request due to an internal error. For more information about the error, either turn on IncludeExceptionDetailInFaults ‎(either from ServiceBehaviorAttribute or from the configuration behavior)‎ on the server in order to send the exception information back to the client, or turn on tracing as per the Microsoft .NET Framework 3.0 SDK documentation and inspect the server trace logs.. --> The server was unable to process the request due to an internal error. For more information about the error, either turn on IncludeExceptionDetailInFaults ‎(either from ServiceBehaviorAttribute or from the configuration behavior)‎ on the server in order to send the exception information back to the client, or turn on tracing as per the Microsoft .NET Framework 3.0 SDK documentation and inspect the server trace logs. 


Looks horrifying right? You have probably tried to use different credentials, created another Migration Endpoint object in Exchange Online, all without avail?

Also you would be scratching your head because you were able to move mailboxes to Exchange Online using the same MRSProxy without issues,

This is probably due to the fact that you have entered the database name wrong:

You probably have databases in a DAG, and when you copied the database name, it ended up something like "database name\server name".

The fact that database names are unique in an Exchange organisation, there is no need to specify the server name.

So what you need to do is enter "database name" into the Target Database field and your migration should be fine.

Office 365 - Exchange Online - Free/Busy Query from Exchange Online Mailbox to On Premises Exchange Fails

This topic is quite a common one, but in my case, the resolution is not.

So let's start with the environment:

• Exchange 2010 SP3 RU12
• TMG 2010 
• HCW v3

Here is the problem: Exchange Online users cannot get free/busy information of users still on premise Exchange. The other way works fine, i.e. Exchange 2010 users can see free/busy information of Exchange Online mailbox users.

First and foremost, you should run through the following tool and checking everything is in place:

https://support.microsoft.com/en-us/help/10092/troubleshooting-free-busy-issues-in-exchange-hybrid-environment

It is a very comprehensive tool, and make sure you don't skip any steps.

In my case, everything checks out in that tool. Even running Office 365 Free/Busy test from Microsoft RCA also returns free/busy data from on premise user:

And the result would be successful, and returns the free/busy data for the on premise user:

As part of the troubleshooting process, you would run the following PowerShell command from Exchange Online:

Test-OrganizationRelationship -UserIdentity onPremMailbox@company.com  -Identity “O365 to On-premises

- 668sscac-01as-41s1-sd21-e5sslsh3d1c5” -Verbose

And the result would be:

STEP 5: Getting organization relationship setting from remote partner…

RESULT: Unable to retrieve organization relationships from remote organization.

RESULT: Error.

LAST STEP: Writing results...

And that's your first indication that something is broken.

If you dig into your Exchange CAS server's IIS log, and search for "testorg", you will see the following entries:

2016-04-07 07:03:12 10.80.5.101 POST /autodiscover/autodiscover.svc - 443 - 10.30.5.47 TestOrganizationRelationship/1.1 200 0 0 109

2016-04-07 07:03:12 10.80.5.101 POST /autodiscover/autodiscover.svc/WSSecurity - 443 - 10.80.5.47 TestOrganizationRelationship/1.1 500 0 0 0

By running the "Test-OrganizationRelationship" PowerShell command, it generates a test against the on premise Exchange server, and although when accessing "/autodiscover/autodiscover.svc" worked (as indicated by HTTP 200 code), but when accessing "/autodiscover/autodiscover.svc/WSSecurity", HTTP error code 500 (internal server error) is record.

OK, we are getting somewhere here. Why is /WSSecurity not accessible?

Let's check our WSSecurity settings on all CAS servers. Go to Exchange Management Shell, run the following commands:

Get-AutodiscoverVirtualDirectory -server | fl *wss*

Get-WebServicesVirtualDirectory -server |  fl *wss*

Notice that the result shows that WSSecurityAuthentication is already set to $true. So what now?

Well, we fixed our problem by setting the flag to $true again.

I know it sounds counter-productive, but apparently setting the flag does something at the backend and actually fixes the problem. 

So execute the following commands from Exchange Management Shell (on premise Exchange):

Get-AutodiscoverVirtualDirectory -server | Set-AutodiscoverVirtualDirectory -WSSecurityAuthentication $true

Get-WebServicesVirtualDirectory -server | Set-WebServicesVirtualDirectory -WSSecurityAuthentication $true

Next, either do a IISReset or just recycle the following AppPools from IIS Manager:

• MSExchangeAutodiscoverAppPool
• MSExchangeServicesAppPool

Voilà! Problem fixed. Run "Test-OrganizationRelationship" from Exchange Online PowerShell and it would work now. And now if you do the free/busy test, it should work.

This took 2 weeks to troubleshoot with 2 different Microsoft engineers, so hopefully this will help someone.

Lync Hybrid, or Exchange Hybrid with hosted UM - CS Static Route is not supported

Learnt this the hard way from two different clients.

Basically, when you need to have Lync Hybrid setup with Skype for Business Online ("SfBO" - same below), or when you are doing Exchange Hybrid with Exchange Online in an environment where there is already Lync setup with Unified Messaging, you have to make sure that there are no CSStaticRoute that exists, or else co-existence will fail.

Symptoms include:

• Lync Hybrid only: On premise Lync users cannot initiate IM with migrated SfBO user
• Exchange Hybrid with Unified Messaging: When calling the user's extension number, the call is directed somewhere else and not to the user's voicemail

This all boils down to the fact that when SharedAddressSpace is configured in Lync on premise with SfBO, CSStaticRoutes are no longer supported, and have to be deleted.

To confirm that you have a static route, run the following from Lync Management Shell:

Get-CsStaticRoutingConfiguration -identity global | Select-Object -ExpandProperty Route

The quickest way to fix this is to issue the following command to delete all static routes:

Set-csstaticroutingconfiguration -identity global -route $null

Enable-CsTopology

Wait for a few minutes and the problem will resolve itself.

OK, you might ask what are the effects of deleting all CSStaticRoutes - it could range from breaking your Polycom RMX integration with Lync to breaking a PSTN conferencing solution.

Unfortunately I have no advice on how to get around with this, other than paying a support ticket to Microsoft and get them to work with the third party vendors.

I was lucky because the customer no longer wants to use the RMX integration, and Polycom RMX integration does not work with Skype for Business Online anyway.

Hope this helps someone.