- Connect with Paul Miller
In this blog, I take you on a journey to the depths of Exchange Hybrid Server upgrades. This niche case may challenge even the most seasoned professionals. If you’re accustomed to working in single domain environments, then this article is a must-read for you. It will help you navigate the pitfalls of supporting a multi-domain deployment and shed light on what’s different when dealing with multiple domains in a single forest. You’ll learn how to uncover mailboxes that may be hidden from view when their corresponding user objects exist only in the root domain/forest level.
Recently, I encountered an upgrade issue with a client who was transitioning from Exchange 2013 to Exchange 2016. While everything seemed to be going smoothly, we hit a snag when we were ready to remove the old servers from the mix. To mitigate any potential problems, I advised the client to shut down the old server for a 7-day period before we began the final decommissioning process. This allowed us ample time to observe any issues that might arise when Exchange is removed from the server and hypervisor.
Fortunately, no immediate issues surfaced when the server was shut down. The new 2016 servers performed as expected, and no clients were affected. It was shaping up to be a success, but things took a turn when administrators attempted to access ECP from the new servers. They were greeted with an error 500, and the ECP page failed to display. This is a telltale sign that the arbitration mailboxes weren’t properly moved from the old database to the new server. Specifically, the mailbox associated with OAB “SystemMailbox{bb558c35-97f1-4cb9-8ff7-d53741dc928c}” is required for access to the EAC page for accounts that don’t have their own mailbox attached.
After some investigation, I confirmed that the issue affecting the administrative accounts could be resolved by adding a standard user mailbox. While this did offer a temporary solution, it’s not the best practice to associate a mailbox with administrative accounts, making it unsuitable as a permanent fix.
I was left scratching my head because I had executed the necessary steps to move all arbitration, monitoring and audit mailboxes to the new servers. To add to the confusion, all databases attached to the old servers appeared to be empty when running the standard get commands. Further exploration was needed to identify the root cause of the issue.
I was unable to locate the OAB system mailbox, SystemMailbox{bb558c35-97f1-4cb9-8ff7-d53741dc928c}. Even an examination of the Exchange System Objects folder, where I anticipated the user account to be located, yielded no results. My quest to uncover the root cause of the issue only deepened as I delved further into the problem.
I discovered that the Get-Mailboxdatabase command piped through the Get-MailboxStatistics command still listed several mailboxes. While most of these were disabled mailboxes and didn’t have any impact on the administrative accounts, a few were still active.
The situation became even more puzzling as I realized that some of the AD accounts associated with the arbitration mailboxes were nowhere to be found on the domain controllers for the specific domain I was working on.
After spending several hours on this the perplexing situation, I uncovered a crucial piece of information: the domain where Exchange was installed, which I was working on, was actually a child domain called “CORP” within a larger parent forest called “root.” This had important implications for the location of the user objects for the arbitration mailboxes because, as a forest-level tool, Exchange requires these objects to be located in the root domain.
Complicating matters further was the fact that the environment had grown organically over time and had undergone multiple iterations of Exchange installations, leading to the objects in question being stored in an unexpected location. Rather than being found in the “Exchange System Objects” container within AD, these objects were actually located in the “Users” container, where they were created via an earlier version of Exchange.
To uncover their whereabouts, I did an AD search using the Active Directory Users and Computers snap-in. I was able to confirm my findings by using the Get-ADUser command with the SamAccountName of the mailbox I was seeking and specifying a domain controller in the root forest.
When working in a multi-domain environment, it’s essential to expand the search scope of Exchange commands to include forest-level objects. However, by default, the search scope is limited to the domain where Exchange is installed. This can create problems when dealing with objects in a parent or child domain. Fortunately, expanding the search scope is a simple task, but one that you need to keep in mind.
To expand the search scope, you can use the Set-ADServerSettings command to ViewEntireForest $true to include forest-level objects in your search. This will enable you to access and work with objects in other domain.
By expanding the search scope with the Set-ADServerSettings command, I was able to locate the system mailboxes and their corresponding user objects in the root domain. Armed with this information, I could now efficiently move the necessary mailboxes to the new Exchange 2016 servers using the New-MoveRequest command. After performing these moves, administrators without mailboxes regained access to the EAC pages, and we were able to decommission the old 2013 databases with ease. This was a major breakthrough, as previous attempts to delete the old databases had resulted in errors.
Don’t let upgrade issues catch you off guard. With the knowledge you’ve gained from this article, you’ll be better equipped to handle multi-domain deployments and ensure a smooth transition to your new Exchange Hybrid Server.
If you need support working through this at your organization, Threadfin can help. Contact us today.
