If Conditional Access behaviour depends on sign-in context, client path, device state, service dependencies, and policy interaction, then static review alone is not enough. Defined scenarios help, but exhaustive deterministic simulation goes further.
Most Conditional Access teams already do some kind of review.
Someone checks the scope. Someone reads the grant controls. Maybe there is a CAB. Maybe the policies are exported and compared in Git. Maybe there are naming standards, baseline templates, and a rough idea of what "good" looks like.
That work is useful.
It is also only the first layer.
Once a Conditional Access estate gets even moderately complex, the real question is no longer just:
It becomes:
Static review cannot answer those questions by itself.
Defined scenario testing gets you much closer. It is a real improvement over policy inspection and one of the best habits a CA team can adopt.
But even that is still sampled coverage.
If your goal is to predict hidden interactions and change impact across a mature Conditional Access estate, the stronger approach is exhaustive deterministic simulation of the effective decision space: not just a few named scenarios, but the whole set of relevant runtime combinations your tenant can actually produce.
That is the distinction this post is about.
A lot of confusion disappears once you stop treating all CA validation as one thing.
There are at least three separate jobs:
They are related, but they do not answer the same question.
Loading diagram…
If a team only has the first layer, it is mostly checking intent.
If it has the first two layers, it is testing representative behaviour.
If it has all three, it can start reasoning about regression, coverage gaps, and change impact with much more confidence.
Static review works well for things it can actually see.
It can catch errors like:
all cloud apps was used where a narrower scope was saferrequire one of the selected controls was chosen where require all was intendedThat is valuable. It is the CA equivalent of linting, config review, or schema validation.
But it still stops at the object boundary.
It tells you that the policy definitions look plausible. It does not reliably tell you what the engine will decide when a real sign-in arrives with a real combination of:
That is the gap.
Conditional Access outcomes are produced by policy plus runtime context. Reviewing the policies alone does not give you a full execution model.
Once teams accept that CA is contextual, the obvious next step is to test scenarios rather than just review objects.
That means describing a concrete sign-in case and asking what the effective outcome should be.
A decent CA test case usually needs at least:
That is already much better than saying "we tested SharePoint" or "the policy looked right."
A scenario like this is meaningful:
That is testable. It has a runtime shape.
This is also where Microsoft’s own tooling points. The Entra What If tool and Microsoft Graph conditionalAccess/evaluate API both exist because administrators need to evaluate policy applicability against sign-in properties, not just stare at policy JSON and hope they have inferred the result correctly.
So yes: finite scenario testing is absolutely better than static review.
It is closer to how engineers test real systems, and it catches classes of failure that object inspection alone will miss.
Defined scenario packs are stronger than static review, but they still have an unavoidable limitation.
They are sampled coverage.
No matter how carefully you choose the cases, you are still testing a finite set of named situations out of a much larger decision space.
That leaves several common blind spots:
This is one reason CA testing often looks stronger on paper than it is operationally. The scenario list may be well written, but it still only samples the estate.
A single browser path for one user on one device is not coverage. Five or ten carefully defined scenarios are better, but they still do not tell you how a policy change reshapes the full effective access surface.
That is where the next layer becomes important.
By exhaustive deterministic simulation, I do not mean random test generation or vague "AI" guessing.
I mean systematically evaluating the effective Conditional Access decision space represented by the tenant model you care about.
In practice, that means taking the relevant dimensions of runtime behaviour, such as:
and evaluating the combinations that are meaningful in the modeled estate.
The output is not just a handful of pass/fail scenario results. It is a coverage map and a decision diff.
You can ask questions like:
That is a stronger capability than running a finite scenario pack, because it is aimed at the shape of the whole effective decision surface rather than a chosen sample from it.
A simple way to picture it is this:
Loading diagram…
The key idea is not "simulate more things" in the abstract.
It is that exhaustive simulation answers a different operational question:
That is what makes it better suited to impact analysis.
This distinction matters more in CA than in a lot of simpler policy systems because CA has several properties that make interaction effects easy to miss:
That is why so many CA incidents sound like this:
Those are not just failures of discipline. They are failures of coverage.
Static review never had a chance of catching them. A finite test pack might catch them if the right scenario happened to be included. Exhaustive simulation is stronger because it is built to expose those interactions systematically.
Microsoft’s built-in tooling is useful evidence here.
The What If tool and Graph evaluation API make it explicit that CA decisions depend on supplied sign-in properties and that accurate evaluation requires as much context as possible. That is already a strong argument against relying on static review alone.
But Microsoft’s tooling also shows the limit of point-in-time scenario evaluation.
The What If tool is excellent for:
It is not, by itself, a full regression framework or an exhaustive impact-analysis engine.
Microsoft’s own documentation also notes an important limitation: the What If tool does not account for Conditional Access service dependencies. That is exactly the kind of hidden interaction that sampled evaluation can miss and that broader simulation work needs to surface.
So the lesson is not that Microsoft’s tooling is weak. It is that point-in-time evaluation and exhaustive impact analysis are different categories of tool.
This is also the fairest way to compare tools like Maester.
Maester is a PowerShell and Pester-based framework for testing and monitoring Microsoft 365 security configuration. It is clearly useful for:
That is real value.
In many teams, adopting something like Maester would be a substantial improvement over ad hoc review, one-off screenshots, and tribal knowledge.
It also has a Conditional Access What-If story, which makes sense for defined scenario checks and change validation.
But that is still not the same thing as exhaustive runtime-decision simulation.
The underlying difference is straightforward:
Maester is well aligned to the first category and can support parts of the second. It is not really aimed at the third in the stronger sense used here.
That is not a criticism of Maester. It is just a category boundary.
If your goal is compliance, guardrails, and repeatable checks against known policy conditions, Maester is a sensible tool.
If your goal is predicting Conditional Access blast radius and hidden interactions across the effective runtime decision space, you need something broader than a finite set of authored tests.
Blast radius is the practical reason this distinction stops being academic.
Suppose you change one CA policy and want to know the impact before enforcement.
Static review can tell you what you edited.
A finite scenario pack can tell you whether your chosen test cases still pass.
Exhaustive simulation can tell you something much closer to what an engineer actually wants to know:
That is a better fit for CA because so many failures are caused by interaction effects rather than obvious syntax errors.
The strongest workflow is not "pick one tool and pretend it does everything."
It is to stack the layers in order:
That gives each layer a clear job.
Imagine a tenant with:
Now change one thing: a broad productivity policy is extended to include another app set.
Three validation layers produce three different kinds of confidence.
Useful, but shallow.
Better.
That is the difference between "the tests we wrote passed" and "we understand the impact of this change."
This is the problem space Altitude Labs is aiming at.
The point is not that static review is useless or that finite scenario testing is bad. Both are worth doing, and many teams still need to get better at them first.
The point is that Conditional Access eventually needs a stronger engineering model than object review plus a small hand-written test pack.
Altitude Labs is better understood in that context: not as a generic compliance checker, but as an approach centred on deterministic simulation, change diffing, regression coverage, and impact analysis across the effective CA decision space.
That is a narrower and more defensible claim than saying it is simply "better" than tools like Maester.
For compliance guardrails and policy-as-code checks, Maester is useful.
For understanding how Conditional Access decisions shift across a tenant when runtime combinations and policy interactions are the real problem, exhaustive deterministic simulation is the stronger tool class.
Once you accept that Conditional Access needs more than static review, the next question is not just how to test one change.
It is how to make the whole thing repeatable.
That means baselines, regression suites, expected-outcome diffs, and eventually cross-tenant modelling.
That is where the series goes next.
Because the real step up is not from policy review to one-off troubleshooting. It is from careful administration to an actual testing discipline.
Next in the series: Regression Testing for Identity Policy: Baselines, Change Impact, and MSP-Scale Conditional Access
Chief Technology Officer writing about AI systems, software architecture, cyber security, cryptography, and the practical realities of technology leadership.