True Story – Avoiding Risk
This past weekend, a medium-sized client contacted us concerning a serious problem. Moments before the call, the admin had unfortunately and inadvertently deleted ALL, YES ALL, Active Directory user objects (excluding service accounts). If you couldn’t have guessed, this was a complete heart stopping experience! At that very moment, I was struck with panic. Unless you’re clued into such technical ramifications, it would be a challenge to truly capture the gravity of this particular situation.
Thankfully, 2 years prior, when upgrading their environment to Active Directory 2008 R2, we took the proper measures to enable their Active Directory recycling bin. This saving grace rescued all of our jobs, and maintained my sanity, allowing for a swift and seamless restoration of these deleted user objects.
As mentioned above; although preventative and proactive measures were previously taken to enable their AD recycling bin, and although straightforward, restoration wasn’t/isn’t as easy as flipping the light switch. Where only a few of us were available for triage, a rapid assessment of the scene assisted greatly to determine our scope of exposure, and immediate action items. With Active Directory replication and Office 365 DirSync services (Azure AD Connect) poised to pull the whole bloody rug out from beneath thousands of unsuspecting end-users, the clock was certainly working against us.
The Active Directory Recycling bin is only available in AD 2008 R2 (or above) forest functional level (FFL). Make no mistake, although the AD recycling bin was available in prior versions of AD, admins weren’t “per se” able to fully restore objects to their original & previous state. Such restorations were inconsistent, fraught with holes and missing attribute values (e.g., back-links to group objects are lost).
Understandably, for those business’ running on a lesser Forest Functional Levels (FFL), due to perhaps change management, compliance, and/or operational reasons, businesses aren’t always able to augment their forest to 2008 or above.
However, if you are not already, we strongly recommend putting in motion the necessary discovery efforts to safely augment your forest to minimally 2008 R2 FFL. If you’re already at 2008 R2 (or greater) FFL, enable the Active Directory Recycling Bin – don’t wait!
Don’t mistakenly think enabling the Active Directory recycling bin is the first and last step. No sir! Combined with enabling your Active Directory recycling bin, ensure to think further by creating a proactive action plan to address the different facets, dependencies and moving elements of such an occurrence:
Documented test controls which can quickly restore deleted object(s) – in some cases, single object restoration is all that’s needed.
- Test your restoration controls
- Validate object functionality (in some cases, the restore object will be computer or service account)
- Within your controls document, identify potential external dependencies and services, (e.g. Remote AD sites, Office 365, Trusts, etc.). This will significantly help to minimize any outage exposure.
- During remediation:
- Temporarily and immediately disable
- Site to Site AD replication
- Temporarily disable any directory sync services (e.g. DirSync, Azure AD Connect, etc.)
- Validate critical services are functioning properly
- Postmortem Review:
- Perform a Root Cause Analysis (RCA) to determine the specifics behind any required object restoration (really dig in to get the details). Don’t treat this matter as a common every-day operational affair.
- Regardless of whether this was an operation-mandated restore, or perhaps something completely unforeseen, be very certain to take the appropriate preventive precautions to ensure:
- The person or process is corrected. (sometimes, identify management processes require remediation)
Review and tighten up of your business’s Least Privileged Access model
- From a security viewpoint, ensure only the appropriate objects were restored. Restoration of privileged (sensitive) objects could result in a serious breach.
Warmest Regards, SHAWN MAY DTS Inc. | Principal Architect & CEO (888) 589-2999 | www.yourDTS.com