Post 3 made the case for deterministic simulation.
That gets you past static review, but it still is not a full operating model.
Once you can simulate Conditional Access decisions, the next step is operational discipline: baselines, versioned expected outcomes, change impact analysis, and regression testing across one tenant or many.
Post 3 made the case for deterministic simulation.
That gets you past static review, but it still is not a full operating model.
Lots of teams can answer a one-off question like this:
Far fewer can answer the questions that show up once the estate gets real:
That is the difference between using simulation as a troubleshooting tool and using it as an engineering discipline.
One-off simulation tells you what happens in a single case. Regression testing tells you whether the estate still behaves the way you intended after change.
That distinction becomes more important as the number of policies, dependencies, and tenants goes up.
A small tenant with a handful of CA policies can get by with careful review, a bit of manual testing, and report-only observation.
That breaks down once you have:
At that point, the problem is no longer "did we check this policy?"
It becomes: do we have a stable definition of expected access behaviour, and can we detect when the estate moves away from it?
That is a regression-testing problem.
A lot of teams keep their baseline at the wrong layer. They baseline policy objects, exports, screenshots, or dashboard views. Those are useful references, but they are not the unit you actually care about.
The unit you care about is the decision outcome for a defined sign-in scenario.
Exact policy identity is a poor baseline for Conditional Access.
Two tenants can implement the same control intent with different:
Even inside one tenant, a sensible refactor can change policy structure without changing the effective access result.
One tenant might express its workforce baseline with one broad policy and two overlays. Another might split the same intent across six narrower policies. If you baseline exact object identity, those tenants look incomparable. If you baseline effective outcomes, they may be functionally aligned.
That is the useful shift.
The baseline is not:
The baseline is closer to this:
That is the layer where regression testing becomes useful.
If you want repeatable testing, you need a catalog of scenarios that actually describes the estate.
Not a vague checklist. Not "test Teams." Not one happy-path sign-in per app.
A useful scenario catalog defines the access situations you care about in a stable, reusable way.
That usually means each scenario has fields like:
For example:
workforce.sharepoint.browser.managed.trusted.freshworkforce.exchange.mobile.unmanaged.untrusted.freshadmin.azureportal.browser.unmanaged.untrusted.freshguest.teams.browser.managed.untrusted.existingregistration.device-enrollment.browser.unmanaged.untrusted.freshThe point is not to invent a cute naming scheme. It is to make the scenarios durable.
A good catalog includes:
This is where many teams still under-test. They cover the common browser flows and stop there, then act surprised when mobile, desktop, bootstrap, or admin recovery paths behave differently.
Once you move past one tenant, raw object identity gets even less useful.
Tenant A may use:
All-Staff-CAHQ NATCA-Base-MFATenant B may use:
SG-WorkforceMelbourne Office EgressGlobal-MFA and SharePoint-Managed-OnlyThose are different objects. They may still represent the same concepts.
That is why cross-tenant regression work needs a common ontology.
In plain English: define the canonical categories first, then map each tenant's real objects into them.
Example categories might look like this:
actor.workforceactor.admin.privilegedactor.guestresource.sharepoint-onlineresource.exchange-onlineresource.admin-surfaceclient.browser-modernclient.mobile-moderndevice.managed-compliantdevice.unmanagedlocation.trusted-egresslocation.untrustedsession.freshThen each tenant maps its own groups, roles, apps, locations, and posture signals into those categories.
Loading diagram…
That is how you compare posture without demanding identical policy construction.
It also helps inside a single tenant. A common ontology forces you to describe what a scenario is before arguing about which policy object happens to implement it today.
Once you have stable scenarios, you need stable expected results.
Those expectations should be stored like any other engineering artifact: versioned, reviewable, and diffable.
The important part is to version the expected outcome, not just the input policy export.
A practical expected-outcome record might capture:
A simple shape might look like this:
[]
[]
This feels boring right up until you need to answer a real question six weeks later.
Then it becomes very useful.
You can see:
That is the basis for real drift detection.
The most useful output from a CA regression system is not a generic "pass."
It is the delta.
You want to know which scenarios changed, how they changed, and whether the change was intended.
That means diffing a candidate state against the current baseline and classifying the result.
Typical deltas worth highlighting include:
That is what blast-radius prediction actually looks like in identity policy work.
Not a generic risk score. Not a red-yellow-green dashboard. A concrete answer to: which scenarios moved?
This is also why regression runs should not be limited to explicit CA edits.
Some of the most consequential changes come from adjacent systems:
If those changes can alter the effective sign-in outcome, they belong in the regression loop.
Once you treat CA as a tested system, rollout gets a lot less theatrical.
The workflow is straightforward:
Loading diagram…
Each step has a distinct job.
Make the policy change, group change, mapping change, or exception change in a candidate state.
Evaluate the scenario catalog against that candidate state.
This is where deterministic simulation earns its keep. You are not waiting for traffic. You are asking the question directly.
Compare the candidate results to the approved baseline.
This is where you see whether the change is narrow and intended or much broader than the author realised.
If five admin scenarios and one enrollment path moved, that is a different conversation from "Teams got an extra MFA prompt from unmanaged devices." The review should be based on deltas, not guesswork.
Report-only is still useful here, just later in the sequence.
It gives you observational feedback from production sign-ins without enforcing the new behaviour. That helps confirm real traffic patterns and catch cases your catalog may not model yet.
But report-only should not be the first serious attempt to understand impact. By that point you should already have a prediction.
This step is easy to skip and worth formalising.
Did production report-only behaviour line up with the predicted scenario diffs?
If not, that tells you something important:
That is not a failure of testing. It is how the test system improves.
Enforcement should be the last step, after the unknowns are reduced to something you are willing to own.
This is the point MSPs often get wrong in both directions.
One mistake is to demand object-level sameness across all customers. That usually collapses on contact with reality. Different tenants have different licensing, app portfolios, risk tolerance, onboarding flows, geography, and exception history.
The opposite mistake is to give up on comparability entirely and settle for dashboards plus local folklore.
That is not a serious operating model either.
The workable middle ground is semantic baselining.
In practice, that means:
That lets an MSP say something useful across customers without pretending every tenant should have the same JSON.
For example, you can ask a cross-tenant question like:
Tenant A may answer with a straight block. Tenant B may answer with phishing-resistant auth plus compliant device. Tenant C may carry a documented temporary exception for a migration window.
Those are different implementations, but they are still comparable outcomes.
That is much more honest than flattening everything into one template or waving away the differences as too hard.
A sensible multi-tenant baseline usually has three layers:
That is enough structure to detect drift without pretending all customers are clones.
A lot of managed identity work gets stuck at the dashboard layer.
Dashboards can tell you things like:
That is operationally useful.
It is also not the same thing as regression testing.
Dashboards are mostly observational.
They depend on real traffic. They are biased toward common paths. They are weak at low-volume or not-yet-observed scenarios. They do not version expected outcomes. They do not tell you how a candidate state compares with the approved baseline before you ship it.
A dashboard can tell you that a sign-in failed yesterday. Regression testing can tell you that a proposed change will alter twelve scenarios, including one enrollment path nobody has exercised this week.
Those are different capabilities.
The same goes for one-off checks in the What If tool. They are useful for examining a case. They are not a regression system until the inputs, expectations, and diffs are structured and repeatable. Microsoft also notes that the What If tool does not test Conditional Access service dependencies, which is exactly the kind of hidden interaction a broader regression model has to account for.
At this point the category line is pretty clear.
This is not just policy review with nicer visuals. It is not a report-only dashboard with a simulation button glued on. And it is not configuration compliance alone, even though configuration checks still have value.
The stronger category is closer to:
That framing is more accurate because it describes the real job.
You are not merely checking whether policy objects look sensible. You are maintaining a tested definition of expected access behaviour and looking for drift when the estate changes.
That is the operational jump.
For teams managing more than one tenant, it is also the only model that scales cleanly without turning every customer into a special case or forcing them into a fake template.
This series started by fixing the mental model. Conditional Access is not a firewall.
Then it moved into why outcomes feel random, why static review runs out of road, and why deterministic simulation belongs between design and rollout.
The last step is to make that repeatable.
The durable model is:
That is what turns Conditional Access from a nervous admin exercise into something closer to engineering.
And for MSP-scale identity work, it is the difference between managing posture and merely watching it.
Chief Technology Officer writing about AI systems, software architecture, cyber security, cryptography, and the practical realities of technology leadership.
session.existing