ADR-007: Multidimensional Classification Matrix and Stop Policy for Iterative Deep Research¶

1. TL;DR¶

We abandon the single weighted "Confidence Index" scalar as the gate for stopping iterative research rounds. Instead, we adopt a DORA-style multidimensional classification matrix where every dimension must independently achieve "High" or better before stopping. The CI scalar is retained only as a tiebreaker for the coalesce() best-of-N selector.

2. Status History¶

3. Context & Problem Statement¶

The agentic-workflows-v2 system runs a 10-node deep research pipeline with bounded iterative rounds R1–R4. Each round produces a research artifact. The system must answer three questions before committing to a round result or continuing:

┌───────────────────────────────────────────────────────────────┐
│           THREE QUESTIONS THE STOP POLICY MUST ANSWER         │
├───────────────────────────────────────────────────────────────┤
│  Q1 │ How should multi-dimensional research quality be scored? │
│  Q2 │ When should iterative refinement stop?                   │
│  Q3 │ How should source temporal freshness factor in?          │
└───────────────────────────────────────────────────────────────┘

3.1 Pipeline Architecture¶

flowchart LR
    subgraph PIPELINE ["🔬 Deep Research Pipeline (R1–R4)"]
        direction LR
        R1["Round 1\nResearch Artifact"] -->|score| E1{Gate?}
        E1 -->|PASS| C["coalesce()\nbest-of-N"]
        E1 -->|FAIL| R2["Round 2\nRefined Artifact"]
        R2 -->|score| E2{Gate?}
        E2 -->|PASS| C
        E2 -->|FAIL| R3["Round 3"]
        R3 -->|score| E3{Gate?}
        E3 -->|PASS| C
        E3 -->|FAIL| R4["Round 4\n(Max)"]
        R4 -->|score| C
    end
    C --> OUT["✅ Final Output"]

3.2 Why the Original CI Formula Failed¶

The original proposal was a weighted arithmetic mean (WAM):

CI = 0.25 × coverage + 0.20 × source_quality + 0.20 × agreement + 0.20 × verification + 0.15 × recency

Stop when: CI ≥ 0.80 AND recent_sources_count ≥ 10 AND critical_contradictions == 0

The compensability problem — a WAM lets high scores in one dimension mask failure in another:

A verification score of 0.20 represents unverified claims — yet the WAM returns a passing score of 0.67. The single composite number hides the failure.

4. Decision¶

Adopt a DORA-style multidimensional classification matrix as the primary stop gate. Demote the CI weighted sum to a secondary tiebreaker role only.

4.1 The Multidimensional Classification Matrix¶

Each research dimension is independently scored into one of four performance tiers:

╔══════════════════════════════════════════════════════════════════╗
║          RESEARCH QUALITY CLASSIFICATION MATRIX                  ║
╠══════════╦═══════════╦════════════╦═══════════╦════════════╦════╣
║ Tier     ║ Coverage  ║ Src Quality║ Agreement ║Verification║Rec.║
╠══════════╬═══════════╬════════════╬═══════════╬════════════╬════╣
║ 🏆 Elite ║  ≥ 0.90   ║  ≥ 0.90   ║  ≥ 0.90   ║  ≥ 0.90   ║≥90d║
║ ✅ High  ║  ≥ 0.75   ║  ≥ 0.75   ║  ≥ 0.75   ║  ≥ 0.75   ║≥60d║
║ ⚠️ Medium║  ≥ 0.50   ║  ≥ 0.50   ║  ≥ 0.50   ║  ≥ 0.50   ║≥30d║
║ ❌ Low   ║  < 0.50   ║  < 0.50   ║  < 0.50   ║  < 0.50   ║<30d║
╚══════════╩═══════════╩════════════╩═══════════╩════════════╩════╝

Dimension definitions:

• Coverage — breadth of topics addressed relative to the query scope
• Source Quality — authority, peer-review status, and citation depth of sources
• Agreement — cross-source corroboration; absence of contradicting claims
• Verification — active fact-checking and claim provenance tracing
• Recency — proportion of sources published within the domain-adaptive window

4.2 Stop Gate Logic¶

flowchart TD
    GS["📊 Score All 5 Dimensions"]
    GS --> CK1{"All dimensions\n≥ High?"}
    CK1 -->|No| CK1F["❌ CONTINUE\nto next round"]
    CK1 -->|Yes| CK2{"recent_sources_count\n≥ 10?"}
    CK2 -->|No| CK2F["❌ CONTINUE\nto next round"]
    CK2 -->|Yes| CK3{"critical_contradictions\n== 0?"}
    CK3 -->|No| CK3F["🛑 HALT (contradiction)\ncoalesce() selects best prior round"]
    CK3 -->|Yes| PASS["✅ STOP\nThis round passes"]

    CK1F --> REG{"Consecutive\nregression\ndetected?"}
    CK2F --> REG
    REG -->|Yes| STOP2["🛑 PATIENCE STOP\ncoalesce() selects best round"]
    REG -->|No| NEXT["▶ Next Round"]

Stopping conditions (formal):

4.3 Best-of-N Tiebreaker (CI as Secondary Scorer)¶

When coalesce() must rank multiple passing rounds — or select the "least bad" round when all fail — the CI weighted sum is used only at this step:

CI (tiebreaker) = 0.25 × coverage
                + 0.20 × source_quality
                + 0.20 × agreement
                + 0.20 × verification
                + 0.15 × recency

⚠️ This weight vector has no empirical calibration. See §7 (Open Questions) for sensitivity analysis requirements.

Alternative tiebreaker weight candidates:

5. Rationale¶

5.1 Production Precedents for Multidimensional Gates¶

5.2 Compensability Danger Illustrated¶

┌─────────────────────────────────────────────────────────┐
│  "The Masking Problem" — WAM Hides Catastrophic Failure  │
│                                                         │
│  Coverage      ████████████████████  0.95              │
│  Src Quality   ████████████████████  0.95              │
│  Agreement     ██████████████████░░  0.90              │
│  Verification  ████░░░░░░░░░░░░░░░░  0.20  ← FAILURE  │
│  Recency       ████████████████████  0.95              │
│                                            ─────────── │
│  WAM Result:   0.79  ✅ (PASSES old CI gate!)          │
│  Matrix Gate:  ❌ FAIL  (Verification = Low)           │
└─────────────────────────────────────────────────────────┘

5.3 Iterative Refinement Degradation — Why Bounds Matter¶

LLM iterative self-improvement has empirically established limits:

xychart-beta
    title "Quality vs. Refinement Round (SELF-REFINE empirical pattern)"
    x-axis ["R1", "R2", "R3", "R4", "R5"]
    y-axis "Relative Quality" 0 --> 130
    line [85, 105, 117, 119, 112]

5.4 Recency — Why 183 Days Is Wrong¶

The prior 183-day freshness window was borrowed from international tax residency law (IRS Substantial Presence Test), not information retrieval. It has no IR basis.

Domain-adaptive window targets (to replace 183-day hard cutoff):

Decision: Replace the fixed 183-day window with a domain-adaptive freshness classifier that infers the appropriate window from source publication rate (following the Google QDF model).

6. Consequences¶

6.1 Positive Outcomes¶

6.2 Trade-offs and Risks¶

7. Open Questions & Required Actions¶

8. Alternatives Considered¶

9. References¶

10. Decision Record Summary¶

┌──────────────────────────────────────────────────────────┐
│                    ADR-007 DECISION MAP                  │
├──────────────────────────────────────────────────────────┤
│                                                          │
│  OLD:  One CI score ──→ threshold gate ──→ stop/continue │
│                                                          │
│  NEW:  5 Dimensions ──→ each classified independently    │
│             │                                            │
│             ├── All ≥ High? ──→ AND ──→ sources ≥ 10?   │
│             │                    AND ──→ contradictions=0│
│             │                     │                     │
│             │                  STOP ✅                   │
│             │                                            │
│             └── Any < High? ──→ CONTINUE or PATIENCE     │
│                                    STOP after regression │
│                                                          │
│  coalesce(): CI score used ONLY to rank candidates       │
└──────────────────────────────────────────────────────────┘

Approved by: (pending review) Next review date: 2026-03-23