AI Governance Framework

01 · The Challenge

The Problem

Organizations adopting AI face a trust gap. Teams don't know when to trust AI outputs. Without structured governance, AI either gets over-relied on (compliance risk) or under-adopted (wasted investment). Every SDLC step — from PRDs to deployment — needs a clear path from full human review to autonomous operation.

The Solution

A 4-phase maturity framework where each SDLC step independently progresses from full human review → confidence-based routing → exception-only review → strategic oversight. Data-driven phase transitions — not gut feel. Each phase has structured review forms that feed a Jira-based governance dashboard.

02 · The 4 Phases

Phase 1

Full Review

Every AI output reviewed by a human. Baseline accuracy, error patterns, confidence calibration established. Reviewer signals when step is ready for Phase 2.

Phase 2

Confidence Routing

High-confidence outputs (≥0.85) get spot-checked. Low-confidence get full review. Validates whether AI confidence score is trustworthy. Reviewer signals when routing is calibrated for Phase 3.

Phase 3

Exception-Based

AI operates autonomously. Humans review only when: AI self-flags, stakeholder escalates, rule-based trigger fires, or random sample selected. Exception validity tracked.

Phase 4

Strategic Oversight

Monthly governance by AI program manager. Program health, regulatory alignment, business value, regression tracking. No per-output review.

03 · Why This Framework

Without Framework

No consistent accuracy tracking

Phase transitions based on gut feel

Compliance gaps discovered after incidents

No data on AI ROI

With Framework

Every output tracked with accuracy + time + errors

Phase transitions backed by 8+ weeks of data

Compliance gaps caught proactively via forms

Clear ROI: hours saved, exception rate, accuracy trends

04 · Architecture Diagram Governance flow across 4 phases

🤖

AI generates outputConfidence score attached (0–1)

Phase 1 — Full Review

All outputs → Human reviewer → Phase 1 Form → Jira (AIGOV)

Phase 2 — Confidence Routing

Confidence ≥0.85 → Spot check | <0.85 → Full review → Phase 2 Form → Jira

Phase 3 — Exception-Based

Autonomous | Exception triggered → Phase 3 Form → Jira

Phase 4 — Strategic Oversight

Monthly governance → Phase 4 Form → Jira

📊

Jira Dashboard (AIGOV)Accuracy trends · Phase progression · Error patterns · Time saved

Phase Progression Criteria

P1 → P2: 8+ weeks of data, accuracy ≥90%, confidence calibration validated, reviewer signals “Ready P2”

P2 → P3: Routing accuracy ≥95%, false negative rate <2%, 8+ weeks in P2, reviewer signals “Ready P3”

P3 → P4: Exception rate <5%, valid exception rate >80%, 12+ weeks in P3, no regressions

Regression triggers: Accuracy drops below threshold, 3+ errors in a week, compliance gap detected

SDLC Steps Covered

Requirements & PRD Planning Architecture Development Code Review Testing Security Deployment Monitoring Reporting

AI Confidence Score — System Prompt & Calibration

How it's generated: Add the following block to every AI system prompt. The model outputs a confidence score on the last line of every response.

After your response, output on the last line EXACTLY this format:
CONFIDENCE: 0.XX

Score between 0.00 and 1.00 based on:
- How well the question maps to your training data and provided context
- Whether you cited specific facts vs made general statements
- Whether you had to speculate or infer beyond the source material
- How complete your answer is relative to what was asked

Scoring guide:
  0.90–1.00  Fully grounded in provided context, specific citations, complete answer
  0.80–0.89  Mostly grounded, minor gaps, high confidence in core claims
  0.70–0.79  Partially grounded, some inference required, notable gaps
  0.60–0.69  Significant inference, limited source material, incomplete
  Below 0.60  Largely speculative, minimal grounding, flag for human review

How the pipeline parses it: The output wrapper strips the last line matching CONFIDENCE: \d\.\d+ and writes it to the Jira ticket as ai_confidence. The line is removed from the user-facing output.

How it's calibrated (Phase 1–2):

Well Calibrated ✓

AI says 0.90 + Human says Correct
AI says 0.70 + Human says Partial
→ Scores align. Threshold is trustworthy.

Miscalibrated ✗

AI says 0.92 + Human says Incorrect
→ AI is overconfident. Lower threshold
or retrain before Phase 2 routing.

Threshold tuning: Default is 0.85. During Phase 2, track false positive rate (good outputs flagged for review) and false negative rate (bad outputs that auto-passed). Adjust threshold until high-confidence outputs achieve ≥95% accuracy on spot checks.

01 · Project Details

Key: AIGOV Name: AI Governance Type: Team-managed Lead: TPM / AI Program Manager

Uses your existing Jira instance (Atlassian Cloud). No new tool needed.

Issue Type Summary — Ticket Volume by Phase

Issue Type	When Created	Who Fills	Volume
P1 Phase 1 Review	Every AI output generated	Any team member reviewing	~24/mo
P2 Phase 2 Review	Spot-check or low-confidence routed	Reviewer assigned by routing	~8/mo
P3 Phase 3 Exception	AI self-flags, stakeholder escalates, rule triggers	Reviewer investigating exception	~2/mo
P4 Phase 4 Governance	Monthly on the 1st (auto-created)	AI Program Manager	1/mo

ROI story: As each SDLC step matures from Phase 1 → 4, ticket volume drops from ~24 human reviews/month to 1 governance check — with data proving every transition.

Custom Fields by Phase

Phase	Total Fields	Inherited	New	New Fields Added
P1 Full Review	14	—	14	output_id, release_name, sdlc_step, output_category, accuracy, correction_type, ai_confidence, human_confidence, review_time, correction_time, error_types, feedback_wrong, feedback_good, phase_signal
P2 Confidence Routing	17	14 from P1	3	routing_reason, routing_validation, threshold_adjust
P3 Exception-Based	10	4 from P1	6	exception_trigger, exception_validity, impact_level, root_cause, remediation, phase_signal
P4 Strategic Oversight	12	—	12	review_month, steps_in_phase4, steps_regressed, quality_rating, exception_rate_pct, valid_exception_pct, regulatory_status, team_fluency, hours_saved, outputs_autonomous, program_status, exec_summary
Total unique fields	35	P1 base (14) + P2 adds 3 + P3 adds 6 + P4 is standalone (12)

02 · 4 Issue Types

Phase 1 Review

Full human review of every AI output. Establishes accuracy baseline and calibrates confidence.

Custom Fields

output_id — unique identifier for the AI output
release_name — release or sprint this output belongs to (e.g. v3.1, Sprint 14, Q3 Launch)
sdlc_step — which SDLC step produced this output
output_category — type of output (PRD, user story, test plan, etc.)
accuracy — Correct | Partial | Incorrect
correction_type — None | Minor | Moderate | Rewrite
ai_confidence — 0–1 score from the AI model
human_confidence — High | Med | Low | None
review_time_min — minutes spent reviewing
correction_time_min — minutes spent correcting
error_types — multi-select: Factual inaccuracy, Missing business context, Incomplete, Wrong tone, Compliance gap, Contradicts prior, Hallucination, Security concern
feedback_wrong — freetext: what went wrong
feedback_good — freetext: what was good
phase_signal — Ready P2 | Not yet | Regression

Open→ In Review→ Corrections Applied→ Closed

Phase 2 Review

Confidence-routed review. Validates whether AI confidence score correlates with actual accuracy.

Custom Fields (all P1 fields plus)

routing_reason — Spot check | Low confidence | System flag | Manual
routing_validation — Correct | False positive | False negative
threshold_adjust — No change | Lower | Raise | Not enough data
phase_signal — Ready P3 | Not yet | Regression→P1

Routed→ Spot Check / Full Review→ Routing Validated→ Closed

Phase 3 Exception

Exception-based review. AI operates autonomously; humans review only flagged or sampled outputs.

Custom Fields

exception_trigger — AI self-flag | Rule-based | Stakeholder | Peer | Random sample | Audit
exception_validity — Valid | False alarm | Missed exception
accuracy — Correct | Partial | Incorrect
impact_level — High | Medium | Low | No impact
root_cause — multi-select: Model drift, Novel input, Regulatory change, Domain gap, Classifier gap, Guardrail miss, Hallucination, Data quality
remediation — Revert P2 | Feed training | Update flags | Escalate | No action
phase_signal — Ready P4 | Not yet | Regression→P2

Exception Filed→ Investigating→ Remediation Set→ Resolved

Phase 4 Governance

Monthly strategic review. Program-level health assessment, regulatory alignment, business value tracking.

Custom Fields

review_month — month being reviewed
steps_in_phase4 — count of SDLC steps at Phase 4
steps_regressed — count of steps that regressed this month
quality_rating — Poor | Fair | Good | Excellent
exception_rate_pct — percentage of outputs that triggered exceptions
valid_exception_pct — percentage of exceptions that were valid
regulatory_status — Gap | Monitoring | Aligned | Leading
team_fluency — Low | Developing | Proficient | Advanced
hours_saved — estimated hours saved vs. manual process
outputs_autonomous — count of outputs handled without human review
program_status — Green | Amber | Red
exec_summary — freetext executive summary

Review Due→ In Progress→ Leadership Briefed→ Filed

03 · Labels

SDLC Labels

requirements planning architecture development code-review testing security deployment monitoring reporting

Category Labels

prd user-story acceptance-criteria tech-spec exec-summary compliance-req

04 · Automation Rules

Auto-create P1 tickets when AI pipeline produces output (POST to Jira API)

Auto-create P4 ticket monthly on the 1st

Alert: accuracy = Incorrect → Slack notification to TPM lead

Alert: 3+ Incorrect in same SDLC step/week → auto-flag regression

Alert: phase_signal = Regression → notify AI program manager

Escalation: P4 program_status = Red → escalate to VP

05 · Dashboard Widgets

Accuracy Trend

Line chart — accuracy by SDLC step over time

Phase Tracker

Which SDLC steps are in which phase

Error Distribution

Bar chart — error types across all reviews

Time Saved

Hours saved vs Phase 1 baseline

Confidence Calibration

AI confidence vs actual accuracy correlation

Exception Rate

Exception rate trend for Phase 3+ steps

Phase 1 — Full Review Form

Purpose: Capture accuracy, errors, and time for every AI output. Build baseline data for confidence calibration.
When to fill out: Every time you review an AI-generated output during Phase 1.

1. Output Details
output_id, release_name, sdlc_step, output_category — identifies what was produced and where

2. Accuracy Rating
Correct / Partial / Incorrect — the core quality signal

3. Correction Type
None / Minor / Moderate / Rewrite — how much effort to fix

4. AI Confidence
Record the model's self-reported confidence (0–1) for calibration tracking

5. Time Tracking
review_time_min + correction_time_min — measures ROI impact

6. Error Classification
Multi-select error types: factual, hallucination, compliance gap, etc.

7. Feedback
What went wrong (freetext) + what was good (freetext)

8. Phase Progression
Ready P2 / Not yet / Regression — reviewer's signal on maturity

Sample Filled Entry

Phase 2 — Confidence Routing Form

Purpose: Validate confidence-based routing. Track whether spot checks and full reviews are correctly assigned.
When to fill out: Every time an output is routed for review (spot check or full review) during Phase 2.

1. Output Details
output_id, release_name, sdlc_step, output_category — same as Phase 1

2. Routing Reason
Spot check / Low confidence / System flag / Manual — why this output was reviewed

3. Routing Validation
Correct / False positive / False negative — was the routing decision right?

4. Accuracy
Correct / Partial / Incorrect — same quality signal as Phase 1

5. Confidence Review
AI confidence + human confidence + threshold_adjust (No change / Lower / Raise / Not enough data)

6. Errors
Same multi-select error classification as Phase 1

7. Feedback
What went wrong + what was good (freetext)

8. Phase Progression
Ready P3 / Not yet / Regression→P1

Sample Filled Entry

Phase 3 — Exception Review Form

Purpose: Document and investigate exceptions when AI operates autonomously. Track root causes and remediation.
When to fill out: When an exception is triggered — AI self-flag, stakeholder escalation, rule-based trigger, or random sample.

1. Output Details
output_id, release_name, sdlc_step, output_category

2. Exception Trigger
AI self-flag / Rule-based / Stakeholder / Peer / Random sample / Audit

3. Exception Validity
Valid / False alarm / Missed exception — was the flag warranted?

4. Accuracy
Correct / Partial / Incorrect

5. Impact Assessment
High / Medium / Low / No impact — business impact if undetected

6. Root Cause
Multi-select: Model drift, Novel input, Regulatory change, Domain gap, Classifier gap, Guardrail miss, Hallucination, Data quality

7. Remediation
Revert P2 / Feed training / Update flags / Escalate / No action

8. Feedback
What went wrong + what was good (freetext)

9. Phase Progression
Ready P4 / Not yet / Regression→P2

Sample Filled Entry

Phase 4 — Strategic Governance Form

Purpose: Monthly program-level health check. Assess regulatory alignment, business value, and regression trends.
When to fill out: Once per month by the AI program manager. Covers all SDLC steps at Phase 4.

1. Review Details
review_month, steps_in_phase4, steps_regressed

2. Program Health Assessment
4 dimensions: quality_rating (Poor–Excellent), exception_rate_pct, valid_exception_pct, program_status (Green/Amber/Red)

3. Regulatory Review
regulatory_status: Gap / Monitoring / Aligned / Leading — compliance posture

4. Novel Scenarios
Document new input types or use cases the AI encountered this month

5. Improvement Priorities
Top 3 areas for model/process improvement next month

6. Business Value
hours_saved, outputs_autonomous, team_fluency (Low–Advanced)

7. Regression Documentation
Steps that regressed, root causes, remediation plans

8. Executive Summary
Freetext narrative for leadership briefing

Sample Filled Entry

Weekly Aggregation Flow

⏰

Every Friday (Scheduled)

weekly_ai_governance_report.py

Step 1 — Query Jira

project = AIGOV AND created >= -7d → pull all tickets from the week

Step 2 — Compute per SDLC Step

Accuracy % Avg review time Top error types Confidence calibration Phase signals Regression flags

Step 3 — Claude Analysis

Aggregated data → Claude API → insights, recommendations, phase transition suggestions

📧 Weekly Email

TPM lead + AI program mgr

💬 Slack Summary

#ai-governance channel

📊 Jira Dashboard

Auto-updated via JQL filters

Sample Report — Requirements & PRD

Week of June 23–27, 2026 · 6 AI-generated PRDs reviewed

83%

Correct

17%

Partial

8 min

Avg review

0.89

Avg AI conf

Output	Release	Accuracy	AI Conf	Review	Errors	Signal
PRD-2026-062	v3.1	Correct	0.91	6 min	—	Ready P2
PRD-2026-063	v3.1	Correct	0.88	7 min	—	Ready P2
PRD-2026-064	v3.1	Correct	0.93	5 min	—	Ready P2
PRD-2026-065	Sprint 14	Partial	0.72	15 min	Missing biz context, Compliance gap	Not yet
PRD-2026-066	Sprint 14	Correct	0.90	6 min	—	Ready P2
PRD-2026-067	Sprint 14	Correct	0.87	8 min	—	Ready P2

Recommendation: Requirements & PRD is ready for Phase 2. 5 of 6 reviews signal "Ready P2". Accuracy 83%, avg confidence 0.89 (well-calibrated). One partial had low confidence (0.72) — correctly self-flagged.

Sample Report — Architecture & Design

Week of June 23–27, 2026 · 4 AI-generated architecture outputs reviewed

50%

Correct

50%

Partial

22 min

Avg review

0.74

Avg AI conf

Output	Release	Accuracy	AI Conf	Review	Errors	Signal
ARCH-2026-018	v3.1	Correct	0.82	15 min	—	Not yet
ARCH-2026-019	v3.1	Partial	0.68	28 min	Missing biz context, Incomplete	Not yet
ARCH-2026-020	Sprint 14	Partial	0.71	25 min	Incomplete, Wrong tone	Not yet
ARCH-2026-021	Sprint 14	Correct	0.76	18 min	—	Not yet

Recommendation: Architecture & Design needs more training. 50% accuracy, avg confidence only 0.74. Top errors: missing business context + incomplete outputs. AI correctly self-flags low confidence on partial results. Stay in Phase 1, focus prompt improvement on system design patterns.

Sample Report — Development & Coding

Week of June 23–27, 2026 · 8 AI-generated code outputs reviewed

75%

Correct

12%

Partial

12%

Incorrect

10 min

Avg review

Output	Release	Accuracy	AI Conf	Review	Errors	Signal
DEV-2026-041	v3.1	Correct	0.94	5 min	—	Ready P2
DEV-2026-042	v3.1	Correct	0.91	7 min	—	Ready P2
DEV-2026-043	v3.1	Correct	0.88	8 min	—	Ready P2
DEV-2026-044	Sprint 14	Correct	0.86	9 min	—	Ready P2
DEV-2026-045	Sprint 14	Correct	0.92	6 min	—	Ready P2
DEV-2026-046	Sprint 14	Correct	0.89	7 min	—	Ready P2
DEV-2026-047	Sprint 14	Partial	0.71	18 min	Security concern	Not yet
DEV-2026-048	Sprint 14	Incorrect	0.65	25 min	Hallucination, Security concern	Regression

Recommendation: Development & Coding mostly strong (75% correct) but flagged one Incorrect with hallucination + security concern. AI confidence was low (0.65) on the incorrect output — correctly self-identified. Stay in Phase 1 for now. Investigate security-related code generation patterns.

⚠ Regression Alert: DEV-2026-048 flagged as Incorrect with hallucinated code and security concern. AI confidence was 0.65. Recommend: add security-focused test cases to prompt, review all AI-generated code touching auth/payment flows with mandatory human review regardless of confidence score.

Sample Report — Code Review

Week of June 23–27, 2026 · 7 AI-generated code review outputs reviewed

71%

Correct

29%

Partial

12 min

Avg review

0.81

Avg AI conf

Output	Release	Accuracy	AI Conf	Review	Errors	Signal
CR-2026-031	v3.1	Correct	0.90	8 min	—	Ready P2
CR-2026-032	v3.1	Correct	0.87	9 min	—	Ready P2
CR-2026-033	v3.1	Partial	0.73	18 min	Missing biz context	Not yet
CR-2026-034	Sprint 14	Correct	0.85	10 min	—	Ready P2
CR-2026-035	Sprint 14	Partial	0.69	20 min	Incomplete, Wrong tone	Not yet
CR-2026-036	Sprint 14	Correct	0.84	9 min	—	Ready P2
CR-2026-037	Sprint 14	Correct	0.82	11 min	—	Ready P2

Recommendation: Code Review trending positive (71% correct, up from 55% last month). Partials have low AI confidence (0.69–0.73) — well-calibrated. Top errors: missing business context in review comments, incomplete coverage of edge cases. Stay in Phase 1 for 2 more weeks then reassess.

Sample Report — Testing & QA

Week of June 23–27, 2026 · 10 AI-generated test outputs reviewed (Phase 2 — confidence routing active)

90%

Correct

10%

Partial

6 min

Avg review

0.91

Avg AI conf

Phase 2 Routing Breakdown

High-conf auto-passed

Spot-checked (of 7)

Low-conf full review

Output	Release	Routing	Accuracy	AI Conf	Routing Valid?	Signal
TEST-2026-081	v3.1	Spot check	Correct	0.93	✓ Correct	Ready P3
TEST-2026-082	v3.1	Auto-passed	—	0.95	—	—
TEST-2026-083	v3.1	Full review	Correct	0.78	False positive	Ready P3
TEST-2026-084	Sprint 14	Auto-passed	—	0.91	—	—
TEST-2026-085	Sprint 14	Spot check	Correct	0.89	✓ Correct	Ready P3
TEST-2026-086	Sprint 14	Full review	Partial	0.74	✓ Correct	Not yet
TEST-2026-087	Sprint 14	Auto-passed	—	0.92	—	—
TEST-2026-088	Sprint 14	Full review	Correct	0.80	False positive	Ready P3
TEST-2026-089	Sprint 14	Auto-passed	—	0.94	—	—
TEST-2026-090	Sprint 14	Auto-passed	—	0.90	—	—

Recommendation: Testing & QA performing excellently in Phase 2. 90% accuracy, confidence well-calibrated (0.91 avg). Routing working correctly — 2 false positives (correct outputs flagged for review unnecessarily) suggest threshold could be raised from 0.85 to 0.88. 4 of 5 reviewed outputs signal "Ready P3".

Phase 2 insight: 7 of 10 outputs auto-passed without human review. Only 5 required human time (2 spot checks + 3 full reviews). Human effort reduced by 50% vs Phase 1 (10 reviews → 5).

Cross-SDLC Weekly Summary

SDLC Step	Reviews	Correct	Avg Time	Avg Conf	Phase	Status
Requirements & PRD	6	83%	8 min	0.89	P1	Ready P2 ↑
Architecture & Design	4	50%	22 min	0.74	P1	Needs training
Development & Coding	8	75%	10 min	0.85	P1	Monitor ⚠
Testing & QA	10	90%	6 min	0.91	P2	Calibrating ✓
Reporting	12	92%	4 min	0.93	P3	Autonomous ✓

Weekly Summary: 40 total AI outputs reviewed. Overall accuracy 78%. Requirements ready to promote to Phase 2. Architecture needs prompt improvement. Development flagged 1 regression (security). Testing and Reporting continue to perform well in higher phases. Hours saved this week vs manual: ~14 hours.

Select Phase Form