HCW v3 Stuck at "Adding Federated Domain"

Exchange Hybrid Configuration Wizard - always interesting times every single time I run this wizard.

My customer has three public facing domains that is required as part of the Exchange Online migration. I've added them into Verified Domain in Office 365 portal, ran HCW v3, added the TXT records required for Microsoft Federation Gateway, and when I clicked on "Verify TXT record" button, all 3 domain changed into "TXT record found", and the first domain proceed to be stuck at "Adding federated domain".

Looking into the HCW log (always a good place to start with %appdata%\Roaming\Microsoft\Exchange Hybrid Configuration\), the following error was found:

2017.04.20 01:08:12.023 *ERROR* [Client=UX, Page=DomainProof, Thread=17] Microsoft.Online.CSE.Hybrid.Provider.PowerShell.PowerShellInvokeException: PowerShell failed to invoke 'Set-FederatedOrganizationIdentifier': Unable to reserve domain "FYDIBOHF25SPDLT.company.com" for Application Identifier "000000004C04A704".  Detailed information: "A Windows Live ID error occurred. Detailed information: "PassportError: Passport error.".". ---> System.Management.Automation.RemoteException: Unable to reserve domain "FYDIBOHF25SPDLT.company.com" for Application Identifier "000000004C04A704".  Detailed information: "A Windows Live ID error occurred. Detailed information: "PassportError: Passport error.".".

                                   --- End of inner exception stack trace ---

                                   at Microsoft.Online.CSE.Hybrid.PowerShell.RemotePowershellSession.RunCommandInternal(Cmdlet cmdlet, SessionParameters parameters, Int32 millisecondsTimeout, PowerShellRetrySettings retrySettings, Boolean skipCmdletLogging)

                                   at Microsoft.Online.CSE.Hybrid.Session.PowerShellOnPremisesSession.SetFederatedOrganizationIdentifier(SmtpDomain accountNamespace, String delegationTrustLink, SmtpDomain defaultDomain)

                                   at Microsoft.Online.CSE.Hybrid.App.ViewModel.Pages.DomainProof.DomainInfo.AddFederatedDomain(IOnPremisesSession session, AppData appData)

                                   at System.Collections.Generic.List`1.ForEach(Action`1 action)

                                   at Microsoft.Online.CSE.Hybrid.App.ViewModel.Pages.DomainProof.VerifyActivity(IOnPremisesSession session, IEnvironment environment)

Restarting HCW did not help.

Resolution: Verify one domain at a time. In the page "Select the domains that you want to be part of your Hybrid Configuration", pick only one domain, and click Next, and proceed to verify that domain. Using the Back button, go back and pick another one (and unselecting the previous one) and verify that domain, and repeat for the last domain.

Hope this helps someone.

Exchange Hybrid Free/Busy - Fails after running HCW to add more domains

So you have decided to run HCW again to include additional domains that you have skipped in the initial setup.

And immediately, you notice that free/busy query from Exchange Online to On Premises have started to fail again.

If you have already fixed it from my previous post, why is this happening again?

Firstly, run the following command from Exchange Online PowerShell:

Test-OrganizationRelationship -UserIdentity onPremiseMailbox@company.com -Identity “O365 to On-premises 6633cadc-0124-4111-2a22-e51f8853d1c5” -Verbose

Note that it will fail at STEP 4:

STEP 4: Getting organization relationship settings from remote partner...

RESULT: Unable to retrieve organization relationships from remote organization.
RESULT: Error.

But if you look back at STEP 3 - you will notice that the target URL is probably showing the new domain that you just added:

STEP 3: Requesting delegation token from the STS...

RESULT: Success.

Retrieved token for target https://autodiscover.newcompany.com/autodiscover/autodiscover.svc/WSSecurity for offer Name=MSExchange

.Autodiscover,Duration=28800(secs)

So what's the problem here? Most likely, this is used as a secondary email address and you haven't bothered to configure autodiscover for it.

To confirm this, run the following command from Exchange Online PowerShell:

Get-OrganizationRelationship | FL

Check out the "TargetAutodiscoverEpr" field, it is probably pointing to https://autodiscover.newcompany.com/autodiscover/autodiscover.svc/WSSecurity, instead of https://autodiscover.company.com/autodiscover/autodiscover.svc/WSSecurity

To solve the problem, either configure autodiscover for that domain (add it in public DNS, and update your TMG rules + add the SAN into your certificate), or just repoint it back to the correct autodiscover URL.

This can be done by executing the following command from Exchange Online PowerShell:

Get-OrganizationRelationship | Set-OrganizationRelationship -TargetAutodiscoverEpr https://autodiscover.company.com/autodiscover/autodiscover.svc/WSSecurity

Office 365 - Exchange Online - Free/Busy Query from Exchange Online Mailbox to On Premises Exchange Fails

This topic is quite a common one, but in my case, the resolution is not.

So let's start with the environment:

• Exchange 2010 SP3 RU12
• TMG 2010 
• HCW v3

Here is the problem: Exchange Online users cannot get free/busy information of users still on premise Exchange. The other way works fine, i.e. Exchange 2010 users can see free/busy information of Exchange Online mailbox users.

First and foremost, you should run through the following tool and checking everything is in place:

https://support.microsoft.com/en-us/help/10092/troubleshooting-free-busy-issues-in-exchange-hybrid-environment

It is a very comprehensive tool, and make sure you don't skip any steps.

In my case, everything checks out in that tool. Even running Office 365 Free/Busy test from Microsoft RCA also returns free/busy data from on premise user:

And the result would be successful, and returns the free/busy data for the on premise user:

As part of the troubleshooting process, you would run the following PowerShell command from Exchange Online:

Test-OrganizationRelationship -UserIdentity onPremMailbox@company.com  -Identity “O365 to On-premises

- 668sscac-01as-41s1-sd21-e5sslsh3d1c5” -Verbose

And the result would be:

STEP 5: Getting organization relationship setting from remote partner…

RESULT: Unable to retrieve organization relationships from remote organization.

RESULT: Error.

LAST STEP: Writing results...

And that's your first indication that something is broken.

If you dig into your Exchange CAS server's IIS log, and search for "testorg", you will see the following entries:

2016-04-07 07:03:12 10.80.5.101 POST /autodiscover/autodiscover.svc - 443 - 10.30.5.47 TestOrganizationRelationship/1.1 200 0 0 109

2016-04-07 07:03:12 10.80.5.101 POST /autodiscover/autodiscover.svc/WSSecurity - 443 - 10.80.5.47 TestOrganizationRelationship/1.1 500 0 0 0

By running the "Test-OrganizationRelationship" PowerShell command, it generates a test against the on premise Exchange server, and although when accessing "/autodiscover/autodiscover.svc" worked (as indicated by HTTP 200 code), but when accessing "/autodiscover/autodiscover.svc/WSSecurity", HTTP error code 500 (internal server error) is record.

OK, we are getting somewhere here. Why is /WSSecurity not accessible?

Let's check our WSSecurity settings on all CAS servers. Go to Exchange Management Shell, run the following commands:

Get-AutodiscoverVirtualDirectory -server | fl *wss*

Get-WebServicesVirtualDirectory -server |  fl *wss*

Notice that the result shows that WSSecurityAuthentication is already set to $true. So what now?

Well, we fixed our problem by setting the flag to $true again.

I know it sounds counter-productive, but apparently setting the flag does something at the backend and actually fixes the problem. 

So execute the following commands from Exchange Management Shell (on premise Exchange):

Get-AutodiscoverVirtualDirectory -server | Set-AutodiscoverVirtualDirectory -WSSecurityAuthentication $true

Get-WebServicesVirtualDirectory -server | Set-WebServicesVirtualDirectory -WSSecurityAuthentication $true

Next, either do a IISReset or just recycle the following AppPools from IIS Manager:

• MSExchangeAutodiscoverAppPool
• MSExchangeServicesAppPool

Voilà! Problem fixed. Run "Test-OrganizationRelationship" from Exchange Online PowerShell and it would work now. And now if you do the free/busy test, it should work.

This took 2 weeks to troubleshoot with 2 different Microsoft engineers, so hopefully this will help someone.

Exchange 2013 Hybrid Configuration Wizard Failing

I couldn't get the Hybrid configuration wizard working. The moment after I sign into Office 365, I get the following error:

HTTP Error 500.19 - Internal Server Error
Absolute physical path "C:\inetpub\custerr\C:\inetpub\custerr" is not allowed in system.webServer/httpErrors section in web.config file. Use relative path instead.

Screenshot here:

Looking up and down the Internet, I could not find any useful clue at all. Out of the whim, I decided to move the mailbox that my admin account is associated with to Exchange 2013 and problem fixed!

Hopefully this is useful for some other poor folks with the same problem.

There are a lot of discussions about Exchange 2013 CU6 causing co-existence and hybrid problems but I do not this has been documented before.