From Auditable to Provable

Part 2: How We Taught the Machine to Collect Its Own Evidence

---

This is Part 2 of our compliance automation series. Part 1 covered how we built an Autonomous Auditor that took our compliance score from 1.5% to 93.5% by making policies explicit, scoped, and enforceable.

---

Part 1 ended with a win.

We had 187 controls marked “Fully Implemented.” Policies were rewritten with mandatory language. Scope covered IT, OT, and cloud. Ownership was assigned. The auditor simulation passed.

Then someone asked the obvious question:

“Can you prove any of this?”

And we realized we had solved the wrong problem.

---

The Limits of Part 1

The system we built in Part 1 was a policy optimization engine. It read documents, found gaps, fixed language, and produced audit-ready artifacts.

But here’s what it couldn’t do:

• It couldn’t query Okta to verify MFA was actually enforced
• It couldn’t pull AWS configs to prove encryption settings
• It couldn’t show a single log entry demonstrating the policy was followed
• It couldn’t answer “when did you last review access?” with anything other than “the policy says quarterly”

We had optimized the documentation. We hadn’t touched reality.

The 93.5% score was real—but it measured policy quality, not operational truth.

---

What Auditors Actually Want

Modern audits don’t run on PDFs. They run on evidence.

NIST frameworks expect machine-generated logs, system configurations, execution records, and attestation interviews. ISO 27001 auditors want to see the ticket where access was revoked, not just the policy saying it should be.

The question isn’t “Do you have a document that says you do this?”

It’s “Can you show me you did it?”

We needed a second system. One that could:

1. Discover where evidence lives
2. Query systems to extract proof
3. Validate that the proof is fresh and complete
4. Package it for auditors
5. Fall back to human attestation when machines can’t reach

So we built it.

---

The Evidence Collection Engine

Part 1 produced Evidence Expectations—definitions of what proof would be needed for each control.

Part 2 turns those expectations into actual evidence.

The architecture has seven layers:

Discovery → Query Generation → Execution → Attestation → Validation → Packaging → Simulation

Each layer has specialized agents. Each agent has one job. Here’s how they work.

---

Layer 1: Discovery

Problem: A control says “enforce MFA for privileged access.” Where does that evidence live?

Solution: The Evidence Source Discovery Agent matches controls to systems.

It consults a System Catalog (Okta, Azure AD, AWS, Splunk, etc.) and a Connector Catalog (which systems have active API integrations). For each control requirement, it identifies:

• Which system holds the evidence
• What type of evidence (logs, configs, reports)
• Whether a connector exists
• Who owns the system

If no connector exists, the control is flagged for attestation fallback. The pipeline doesn’t stop—it routes to humans.

---

Layer 2: Query Generation

Problem: We know Okta has the evidence. What query extracts it?

Solution: Three agents generate executable queries:

Positive Test Generator — Creates the query that proves the claim is true.

"Show me all privileged sessions where MFA was used"
→ Okta API: GET /logs?filter=eventType eq "user.session.start" 
  AND actor.id in (privileged_users) 
  AND authenticationContext.authenticationStep ge 2

Counterfactual Test Generator — Creates the query that would disprove the claim.

"Show me any privileged session where MFA was NOT used"
→ Same query, but authenticationStep lt 2
→ Expected result: 0 records

Config Export Generator — Creates commands for static configuration evidence.

"Export the Azure AD password policy"
→ GET /policies/authenticationMethodsPolicy
→ Expected: minLength >= 14

The counterfactual is critical. Every claim gets a test for its opposite. If you can’t run the query that would find violations, you can’t prove the control works.

---

Layer 3: Execution

Problem: We have queries. Now run them.

Solution: The Evidence Collector Agent executes queries against real systems (or mocks for testing).

It handles:

• Authentication via connector
• Pagination for large result sets
• Retry with backoff on failures
• SHA-256 hashing of results (chain of custody)
• Staging to disk with timestamps

If collection fails after retries, the requirement routes to attestation.

Output:

json

{
  "requirement_id": "AC-2.Q1",
  "collection_status": "success",
  "record_count": 1500,
  "pages_retrieved": 3,
  "staged_file": "evidence_staging/AC-2/AC-2.Q1/auth_logs_20260116.json",
  "sha256_hash": "a1b2c3d4...",
  "collection_timestamp": "2026-01-16T01:18:53Z"
}

---

Layer 4: Attestation (The Human Fallback)

Problem: Not everything has an API. Legacy systems exist. Some evidence is procedural.

Solution: When machine collection fails, the Attestation Scheduler creates a Zendesk ticket.

But not a generic ticket. A structured attestation request:

markdown

## Attestation Required: AC-2.Q3

**Requirement:** Inactive accounts disabled within 30 days
**Reason:** Connector unavailable for legacy HR system
**Deadline:** 2026-01-22

### Questions to Address
1. Walk me through the process for identifying inactive accounts.
2. Show me the report you run to find them.
3. How frequently is this executed?
4. Show me an example of an account disabled via this process.

### Artifacts Requested
- [ ] Screen recording of process execution
- [ ] Sample report output

The questions are generated dynamically based on the control requirement. The owner responds with comments and attachments.

An Attestation Reviewer Agent then validates the response:

• Were all artifacts provided?
• Were all questions addressed?
• Completeness score: 0-100%

If incomplete, it adds a follow-up comment requesting the missing items. If complete, the attestation becomes evidence.

---

Layer 5: Validation

Problem: We have evidence. Is it good enough?

Solution: Three validators run before packaging:

Freshness Validator — Is the evidence recent enough?

• Policy requires evidence within 90 days
• Evidence collected 77 days ago
• Status: EXPIRING_SOON (warning)
• Action: Schedule re-collection

Completeness Validator — Do we have all required evidence types?

• Requirement expects: auth_logs + config_export
• Collected: auth_logs + config_export
• Coverage: 100%
• Status: PASS

Counterfactual Analyzer — Did the negative test find violations?

• Query found 3 sessions without MFA
• Severity: HIGH
• Status: POTENTIAL_ISSUE
• Sample violations extracted for remediation

These roll up into an overall validation status: PASS, NEEDS_WORK, or FAIL.

Only evidence that passes validation gets packaged.

---

Layer 6: Packaging

Problem: Auditors need organized, verifiable evidence bundles.

Solution: The Evidence Packager creates a self-contained folder:

output/evidence_packs/AC-2/
├── Manifest.json           ← Hashes, timestamps, metadata
├── Evidence_Index.md       ← Human-readable summary
├── Validation_Summary.json ← Pass/fail details
├── Tests/
│   ├── positive_test.json      ← Query that was run
│   └── counterfactual_test.json
├── Evidence/
│   ├── auth_logs_20260116.json
│   └── config_export_20260116.json
├── Attestations/
│   └── ticket_1001.json    ← If applicable
└── Attestation.md          ← Sign-off template

The Manifest includes SHA-256 hashes of every file. If anything is tampered with, the hash won’t match.

The Evidence Index is what auditors actually read:

markdown

# Evidence Bundle: AC-2 Account Management

**Validation Status:** ✅ PASS

| Requirement | Evidence Type | Status | Freshness |
|-------------|---------------|--------|-----------|
| AC-2.Q1 | auth_logs | ✅ Collected | Valid (88 days) |
| AC-2.Q1 | config_export | ✅ Collected | Valid (88 days) |
| AC-2.Q2 | attestation | ✅ Approved | Ticket #1001 |

## Counterfactual Test Results
**Test:** Find privileged sessions without MFA
**Result:** ✅ 0 violations found

---

Layer 7: Simulation

Problem: Will this evidence survive a real audit?

Solution: The Auditor Simulation Agent plays devil’s advocate.

It loads a playbook of common auditor challenges:

yaml

AC:
  challenges:
    - question: "Show me the process for disabling terminated employees"
      evidence_needed: ["termination_procedure", "hr_iam_log"]
    - question: "What's the SLA for access revocation?"
      evidence_needed: ["standard_with_sla"]
    - question: "How do you handle emergency access?"
      evidence_needed: ["emergency_access_procedure", "emergency_access_log"]

For each challenge, it checks if the bundle has the required evidence. Then it issues an opinion:

Pass Rate	Opinion
100%	UNQUALIFIED_OPINION — Evidence sufficient
80-99%	QUALIFIED_OPINION — Minor gaps noted
50-79%	QUALIFIED_OPINION — Material gaps
<50%	ADVERSE_OPINION — Insufficient evidence

If gaps exist, it generates specific recommendations:

json

{
  "issue": "No emergency access log in evidence bundle",
  "action": "Collect emergency access events from Okta admin logs or schedule attestation with IAM team"
}

Fix the gaps before the real auditor arrives.

---

The Orchestrator

All seven layers are wired together by the Evidence Collection Facilitator:

bash

# Single control
python -m src.run_production_audit --control AC-2

# Entire family
python -m src.run_production_audit --family AC

# All controls, dry run (discover only)
python -m src.run_production_audit --all --dry-run

It handles:

• Happy path: Discovery → Query → Execute → Validate → Package → Simulate
• Fallback path: Discovery fails → Attestation → Validate → Package
• Failure path: Validation fails → Block packaging, report issues
• Batch processing: Mixed results don’t crash the pipeline

Output: pipeline_report.json (machine-readable) and audit_report.md (human-readable).

---

What Changed From Part 1 to Part 2

Part 1	Part 2
Optimized policies	Collected evidence
Score based on documentation	Score based on proof
“We have a policy”	“Here are the logs”
Auditor reads PDFs	Auditor verifies hashes
Attestation = checkbox	Attestation = recorded interview
Trust the document	Run the counterfactual

The 93.5% from Part 1 was policy compliance.

Part 2 answers: Can you prove it?

---

The Results

We ran the evidence collection pipeline against the same 200 controls from Part 1.

Collection Summary:

• 162 controls: Evidence collected via API
• 25 controls: Evidence via attestation (Zendesk)
• 13 controls: Awaiting attestation response

Validation Summary:

• 168 controls: PASS
• 19 controls: NEEDS_WORK (expiring soon or partial)
• 13 controls: AWAITING_ATTESTATION

Simulation Summary:

• 152 controls: UNQUALIFIED_OPINION
• 16 controls: QUALIFIED_OPINION
• 0 controls: ADVERSE_OPINION

Every packaged control now has:

• Machine-collected evidence with hashes
• Counterfactual test results
• Validation timestamps
• Auditor-ready bundle

---

The Counterfactual Matters

The most important addition in Part 2 is the counterfactual test.

Part 1 could verify that a policy said “MFA required.”

Part 2 can verify that no privileged session succeeded without MFA in the last 90 days.

That’s the difference between documentation and proof.

If the counterfactual query returns results, you don’t have a compliant control—you have a finding. The system captures sample violations and routes them to remediation.

This is how auditors think. Now the system thinks that way too.

---

What We Learned

1. Evidence collection is the real work

Fixing policies took 45 minutes. Building the evidence pipeline took 8 sprints. The ratio tells you where the value is.

2. Fallback paths are not optional

38 controls (19%) required attestation because APIs didn’t exist or connectors weren’t available. A system that only handles the happy path handles 81% of reality.

3. Validation prevents garbage

13 controls were blocked from packaging because evidence was stale or incomplete. Without the validation gate, we’d have shipped bad evidence to auditors.

4. Simulation catches gaps before auditors do

16 controls got QUALIFIED_OPINION due to missing evidence types. We found those gaps internally, not during the audit.

---

The Full Picture

Two systems, one goal:

Part 1: Truth → Fix → Proof → Reality Make policies auditable

Part 2: Discover → Query → Collect → Validate → Package → Simulate Make compliance provable

Part 1 answers: “Is the policy explicit, scoped, and enforceable?”

Part 2 answers: “Can we demonstrate it’s actually happening?”

Together, they close the loop between what we say and what we can prove.

---

What’s Next

The evidence collection pipeline runs on-demand today. The next evolution:

• Continuous collection — Scheduled runs that keep evidence fresh
• Expiration alerts — Proactive warnings before evidence goes stale
• Real connectors — Production integrations beyond mocks
• Dashboard — Real-time visibility into evidence status across all controls

The goal remains the same:

Compliance that proves itself.

Not once a year. Not when auditors ask. Continuously.

---

The Takeaway

Part 1 taught us that compliance is a systems problem, not a document problem.

Part 2 taught us that proof is an engineering problem, not a policy problem.

The auditor’s question—”Can you prove it?”—now has a concrete answer:

Yes. Here’s the evidence bundle. The hashes are in the manifest. The counterfactual found zero violations. The simulation passed.

That’s not compliance theater.

That’s compliance as code.

---

— Fiber Dan

---

Read Part 1: From Paper Compliance to Operational Truth