Strengthen chapter review quality gates

- chapter_review.yml: mandatory PROSE EVIDENCE section (3-5 verbatim quotes)
- CHARACTER VOICE AUDIT with per-constraint checks and violation quoting
- Score anchors (95-100/85-94/70-84/<70) to break rubber-stamp clustering
- MUST-FIX items require ORIGINAL quote + PROBLEM + FIX rewrite
- Adjudication: prose_evidence(35) specificity(30) voice_audit(20) coverage(15)
- chapter_roundtable.yml: require quoted evidence, add character_profiles check
This commit is contained in:
David Baity
2026-04-01 22:33:53 -04:00
parent c5c3ca8440
commit 9c362b92f8
2 changed files with 82 additions and 44 deletions

View File

@@ -1,7 +1,8 @@
name: chapter_review
description: >
Independent chapter review. Agent reads the chapter and produces a structured
critique in their editorial domain with explicit must-fix vs optional categorization.
critique with mandatory verbatim prose quotes, score-anchored verdict, and
explicit character voice violation checks.
debug: true
rag_exclude: [business_plan, charter]
model: power
@@ -18,7 +19,7 @@ sections:
steps:
- type: think
max_tokens: 4000
max_tokens: 5000
hint: |
TARGET AUDIENCE: {genre_audience}
GENRE: {genre_name}
@@ -29,54 +30,82 @@ steps:
{chapter_text}
---
Write a detailed editorial review from your perspective.
Be specific — cite line numbers or quote passages where relevant.
Structure your review using EXACTLY these six labeled sections:
Write a detailed editorial review. You MUST quote the actual text to support every
claim you make. Vague commentary ("the pacing drags") without a quoted passage is
not acceptable and will cause this review to fail adjudication.
1. STRENGTHS TO PRESERVE
List elements that are working well and must survive the editorial process unchanged.
Be specific: quote or paraphrase the passage. Vague praise ("the writing is good")
is not useful here.
ALSO CHECK VOICE SIGNATURES:
CHARACTER VOICE PROFILES:
Structure your review using EXACTLY these seven labeled sections:
1. PROSE EVIDENCE
Quote 3-5 passages verbatim from this chapter (use quotation marks and indicate
approximate position: "early", "mid", "late").
For each quote, give one sentence of inline commentary: what it does well or
what it fails to do. This section is MANDATORY -- do not summarize or paraphrase.
A review with zero verbatim quotes will automatically score below 70.
2. CHARACTER VOICE AUDIT
{character_profiles}
If the above block is non-empty: verify each named character in this chapter
speaks to their profile. Can you identify each character's dialogue without
speaker tags? Note YES or NO for each main character.
If the block is empty: check RAG/project context for "## Voice Signatures"
and note whether voice differentiation is present.
If the above block is non-empty: for EACH named character who speaks in this
chapter, check their voice against their profile. Quote one line of their
dialogue and state YES or NO for each constraint:
- Does the character use their signature vocabulary / verbal tics?
- Do they avoid any explicitly forbidden speech patterns?
- Is their emotional register consistent with their arc position?
If a violation is found, quote the offending line and state exactly what rule
is broken (e.g. "Dorian uses contraction 'don't' -- profile forbids contractions
unless in extreme pain or physical exhaustion. This scene does not qualify.").
If the block is empty: check RAG context for "## Voice Signatures" and report
what you find.
2. MUST-FIX — CONTINUITY
3. STRENGTHS TO PRESERVE
List 2-4 elements working well that must survive the editorial process unchanged.
Each item must include a verbatim quote or specific scene reference.
Generic praise ("the atmosphere is strong") without evidence is not allowed here.
4. MUST-FIX -- CONTINUITY
List factual errors, POV breaks, timeline inconsistencies, or world-rule violations.
These are non-negotiable — they cannot be left as "optional." For each item, state:
- What the error is
- What the correction should be
These are non-negotiable. For each item state:
- ORIGINAL: quote the offending passage verbatim
- PROBLEM: what rule or established fact it violates
- FIX: the concrete correction (rewrite the sentence if necessary)
3. MUST-FIX CLARITY
5. MUST-FIX -- CLARITY
List passages where meaning is genuinely obscured, transitions are dropped, or
threads are left dangling in a way that blocks reader comprehension. For each item:
- Quote or reference the passage
- State the concrete fix
- ORIGINAL: quote the offending passage verbatim
- PROBLEM: why it blocks comprehension
- FIX: the concrete correction
4. OPTIONAL SUGGESTIONS
6. OPTIONAL SUGGESTIONS
Craft improvements that would strengthen the chapter but are NOT required for it to
pass. Label each suggestion clearly as optional. Do not inflate this section — only
include suggestions with a clear upside and low risk of voice damage.
⚠️ Do NOT suggest adding thematic contrast metaphors, making dialogue more elegant,
or smoothing sentence variation — these are voice PRESERVATION issues, not fixes.
pass. Label each suggestion clearly as optional. Include the relevant quote.
Do not inflate this section -- only include suggestions with a clear upside and low
risk of voice damage.
Do NOT suggest adding thematic contrast metaphors, making dialogue more elegant,
or smoothing sentence variation -- these are voice PRESERVATION issues, not fixes.
5. FORBIDDEN CHANGES / NON-GOALS
List things that might appear to be problems but should NOT be changed intentional
voice choices, genre conventions, structural decisions that are working. This section
protects the draft from over-editing.
ALWAYS list here: verbal tics, repeated phrases, and "imperfect" speech that is
7. FORBIDDEN CHANGES / NON-GOALS
List things that might appear to be problems but should NOT be changed -- intentional
voice choices, genre conventions, structural decisions that are working.
ALWAYS include: verbal tics, repeated phrases, and "imperfect" speech that is
clearly a character signature, not an error.
6. VERDICT
8. VERDICT
One of: PASS / REVISE
SCORE ANCHORS -- be honest, scores that cluster at 90-96 without strong evidence
signal a rubber-stamp review and will be rejected by the adjudicator:
95-100: Zero MUST-FIX items. PROSE EVIDENCE quotes show above-average craft.
Voice audit finds no violations.
85-94: 1-2 minor MUST-FIX items. At least one PROSE EVIDENCE quote shows
a clear weakness with explanation.
70-84: Clear structural or voice problems. Every MUST-FIX item is quoted
verbatim with a rewrite suggestion.
Below 70: Multiple serious violations. Every failing passage is quoted and
corrected. Automatic REVISE verdict.
State your score and a one-sentence justification citing your evidence.
Use PASS if only optional improvements remain.
Use REVISE if any MUST-FIX items are present.
Reserve judgment on full rewrites — that decision belongs to the roundtable.
- type: document
filename: "Chapter_{chapter_number}_review_{review_letter}"
@@ -89,12 +118,16 @@ adjudication:
pass_threshold: 60
deliverable_type: editorial_review
criteria:
specificity:
weight: 40
description: "Review cites specific passages, scenes, or structural elements in all sections — not vague general commentary"
actionability:
prose_evidence:
weight: 35
description: "Each MUST-FIX item includes a concrete, implementable correction; OPTIONAL items are clearly labeled"
description: "Review contains 3+ verbatim quoted passages with inline commentary. Zero quotes = automatic fail. Paraphrases do not count as quotes."
specificity:
weight: 30
description: "Every MUST-FIX item includes the original passage quoted verbatim AND a concrete rewrite or correction. Generic commentary without quotes scores zero here."
voice_audit:
weight: 20
description: "CHARACTER VOICE AUDIT section checks each speaking character against their profile constraints. Each violation is quoted and the broken rule named explicitly."
coverage:
weight: 25
description: "All six sections are present; STRENGTHS and FORBIDDEN sections are populated, not left empty"
weight: 15
description: "All eight sections present and populated. STRENGTHS section includes evidence. FORBIDDEN section is not left empty."