Strengthen chapter review quality gates
- chapter_review.yml: mandatory PROSE EVIDENCE section (3-5 verbatim quotes) - CHARACTER VOICE AUDIT with per-constraint checks and violation quoting - Score anchors (95-100/85-94/70-84/<70) to break rubber-stamp clustering - MUST-FIX items require ORIGINAL quote + PROBLEM + FIX rewrite - Adjudication: prose_evidence(35) specificity(30) voice_audit(20) coverage(15) - chapter_roundtable.yml: require quoted evidence, add character_profiles check
This commit is contained in:
David Baity
@@ -1,7 +1,8 @@
|
|||||||
name: chapter_review
|
name: chapter_review
|
||||||
description: >
|
description: >
|
||||||
Independent chapter review. Agent reads the chapter and produces a structured
|
Independent chapter review. Agent reads the chapter and produces a structured
|
||||||
critique in their editorial domain with explicit must-fix vs optional categorization.
|
critique with mandatory verbatim prose quotes, score-anchored verdict, and
|
||||||
|
explicit character voice violation checks.
|
||||||
debug: true
|
debug: true
|
||||||
rag_exclude: [business_plan, charter]
|
rag_exclude: [business_plan, charter]
|
||||||
model: power
|
model: power
|
||||||
@@ -18,7 +19,7 @@ sections:
|
|||||||
|
|
||||||
steps:
|
steps:
|
||||||
- type: think
|
- type: think
|
||||||
max_tokens: 4000
|
max_tokens: 5000
|
||||||
hint: |
|
hint: |
|
||||||
TARGET AUDIENCE: {genre_audience}
|
TARGET AUDIENCE: {genre_audience}
|
||||||
GENRE: {genre_name}
|
GENRE: {genre_name}
|
||||||
@@ -29,54 +30,82 @@ steps:
|
|||||||
{chapter_text}
|
{chapter_text}
|
||||||
---
|
---
|
||||||
|
|
||||||
Write a detailed editorial review from your perspective.
|
Write a detailed editorial review. You MUST quote the actual text to support every
|
||||||
Be specific — cite line numbers or quote passages where relevant.
|
claim you make. Vague commentary ("the pacing drags") without a quoted passage is
|
||||||
Structure your review using EXACTLY these six labeled sections:
|
not acceptable and will cause this review to fail adjudication.
|
||||||
|
|
||||||
1. STRENGTHS TO PRESERVE
|
Structure your review using EXACTLY these seven labeled sections:
|
||||||
List elements that are working well and must survive the editorial process unchanged.
|
|
||||||
Be specific: quote or paraphrase the passage. Vague praise ("the writing is good")
|
1. PROSE EVIDENCE
|
||||||
is not useful here.
|
Quote 3-5 passages verbatim from this chapter (use quotation marks and indicate
|
||||||
ALSO CHECK VOICE SIGNATURES:
|
approximate position: "early", "mid", "late").
|
||||||
CHARACTER VOICE PROFILES:
|
For each quote, give one sentence of inline commentary: what it does well or
|
||||||
|
what it fails to do. This section is MANDATORY -- do not summarize or paraphrase.
|
||||||
|
A review with zero verbatim quotes will automatically score below 70.
|
||||||
|
|
||||||
|
2. CHARACTER VOICE AUDIT
|
||||||
{character_profiles}
|
{character_profiles}
|
||||||
If the above block is non-empty: verify each named character in this chapter
|
If the above block is non-empty: for EACH named character who speaks in this
|
||||||
speaks to their profile. Can you identify each character's dialogue without
|
chapter, check their voice against their profile. Quote one line of their
|
||||||
speaker tags? Note YES or NO for each main character.
|
dialogue and state YES or NO for each constraint:
|
||||||
If the block is empty: check RAG/project context for "## Voice Signatures"
|
- Does the character use their signature vocabulary / verbal tics?
|
||||||
and note whether voice differentiation is present.
|
- Do they avoid any explicitly forbidden speech patterns?
|
||||||
|
- Is their emotional register consistent with their arc position?
|
||||||
|
If a violation is found, quote the offending line and state exactly what rule
|
||||||
|
is broken (e.g. "Dorian uses contraction 'don't' -- profile forbids contractions
|
||||||
|
unless in extreme pain or physical exhaustion. This scene does not qualify.").
|
||||||
|
If the block is empty: check RAG context for "## Voice Signatures" and report
|
||||||
|
what you find.
|
||||||
|
|
||||||
2. MUST-FIX — CONTINUITY
|
3. STRENGTHS TO PRESERVE
|
||||||
|
List 2-4 elements working well that must survive the editorial process unchanged.
|
||||||
|
Each item must include a verbatim quote or specific scene reference.
|
||||||
|
Generic praise ("the atmosphere is strong") without evidence is not allowed here.
|
||||||
|
|
||||||
|
4. MUST-FIX -- CONTINUITY
|
||||||
List factual errors, POV breaks, timeline inconsistencies, or world-rule violations.
|
List factual errors, POV breaks, timeline inconsistencies, or world-rule violations.
|
||||||
These are non-negotiable — they cannot be left as "optional." For each item, state:
|
These are non-negotiable. For each item state:
|
||||||
- What the error is
|
- ORIGINAL: quote the offending passage verbatim
|
||||||
- What the correction should be
|
- PROBLEM: what rule or established fact it violates
|
||||||
|
- FIX: the concrete correction (rewrite the sentence if necessary)
|
||||||
|
|
||||||
3. MUST-FIX — CLARITY
|
5. MUST-FIX -- CLARITY
|
||||||
List passages where meaning is genuinely obscured, transitions are dropped, or
|
List passages where meaning is genuinely obscured, transitions are dropped, or
|
||||||
threads are left dangling in a way that blocks reader comprehension. For each item:
|
threads are left dangling in a way that blocks reader comprehension. For each item:
|
||||||
- Quote or reference the passage
|
- ORIGINAL: quote the offending passage verbatim
|
||||||
- State the concrete fix
|
- PROBLEM: why it blocks comprehension
|
||||||
|
- FIX: the concrete correction
|
||||||
|
|
||||||
4. OPTIONAL SUGGESTIONS
|
6. OPTIONAL SUGGESTIONS
|
||||||
Craft improvements that would strengthen the chapter but are NOT required for it to
|
Craft improvements that would strengthen the chapter but are NOT required for it to
|
||||||
pass. Label each suggestion clearly as optional. Do not inflate this section — only
|
pass. Label each suggestion clearly as optional. Include the relevant quote.
|
||||||
include suggestions with a clear upside and low risk of voice damage.
|
Do not inflate this section -- only include suggestions with a clear upside and low
|
||||||
⚠️ Do NOT suggest adding thematic contrast metaphors, making dialogue more elegant,
|
risk of voice damage.
|
||||||
or smoothing sentence variation — these are voice PRESERVATION issues, not fixes.
|
Do NOT suggest adding thematic contrast metaphors, making dialogue more elegant,
|
||||||
|
or smoothing sentence variation -- these are voice PRESERVATION issues, not fixes.
|
||||||
|
|
||||||
5. FORBIDDEN CHANGES / NON-GOALS
|
7. FORBIDDEN CHANGES / NON-GOALS
|
||||||
List things that might appear to be problems but should NOT be changed — intentional
|
List things that might appear to be problems but should NOT be changed -- intentional
|
||||||
voice choices, genre conventions, structural decisions that are working. This section
|
voice choices, genre conventions, structural decisions that are working.
|
||||||
protects the draft from over-editing.
|
ALWAYS include: verbal tics, repeated phrases, and "imperfect" speech that is
|
||||||
ALWAYS list here: verbal tics, repeated phrases, and "imperfect" speech that is
|
|
||||||
clearly a character signature, not an error.
|
clearly a character signature, not an error.
|
||||||
|
|
||||||
6. VERDICT
|
8. VERDICT
|
||||||
One of: PASS / REVISE
|
One of: PASS / REVISE
|
||||||
|
SCORE ANCHORS -- be honest, scores that cluster at 90-96 without strong evidence
|
||||||
|
signal a rubber-stamp review and will be rejected by the adjudicator:
|
||||||
|
95-100: Zero MUST-FIX items. PROSE EVIDENCE quotes show above-average craft.
|
||||||
|
Voice audit finds no violations.
|
||||||
|
85-94: 1-2 minor MUST-FIX items. At least one PROSE EVIDENCE quote shows
|
||||||
|
a clear weakness with explanation.
|
||||||
|
70-84: Clear structural or voice problems. Every MUST-FIX item is quoted
|
||||||
|
verbatim with a rewrite suggestion.
|
||||||
|
Below 70: Multiple serious violations. Every failing passage is quoted and
|
||||||
|
corrected. Automatic REVISE verdict.
|
||||||
|
|
||||||
|
State your score and a one-sentence justification citing your evidence.
|
||||||
Use PASS if only optional improvements remain.
|
Use PASS if only optional improvements remain.
|
||||||
Use REVISE if any MUST-FIX items are present.
|
Use REVISE if any MUST-FIX items are present.
|
||||||
Reserve judgment on full rewrites — that decision belongs to the roundtable.
|
|
||||||
|
|
||||||
- type: document
|
- type: document
|
||||||
filename: "Chapter_{chapter_number}_review_{review_letter}"
|
filename: "Chapter_{chapter_number}_review_{review_letter}"
|
||||||
@@ -89,12 +118,16 @@ adjudication:
|
|||||||
pass_threshold: 60
|
pass_threshold: 60
|
||||||
deliverable_type: editorial_review
|
deliverable_type: editorial_review
|
||||||
criteria:
|
criteria:
|
||||||
specificity:
|
prose_evidence:
|
||||||
weight: 40
|
|
||||||
description: "Review cites specific passages, scenes, or structural elements in all sections — not vague general commentary"
|
|
||||||
actionability:
|
|
||||||
weight: 35
|
weight: 35
|
||||||
description: "Each MUST-FIX item includes a concrete, implementable correction; OPTIONAL items are clearly labeled"
|
description: "Review contains 3+ verbatim quoted passages with inline commentary. Zero quotes = automatic fail. Paraphrases do not count as quotes."
|
||||||
|
specificity:
|
||||||
|
weight: 30
|
||||||
|
description: "Every MUST-FIX item includes the original passage quoted verbatim AND a concrete rewrite or correction. Generic commentary without quotes scores zero here."
|
||||||
|
voice_audit:
|
||||||
|
weight: 20
|
||||||
|
description: "CHARACTER VOICE AUDIT section checks each speaking character against their profile constraints. Each violation is quoted and the broken rule named explicitly."
|
||||||
coverage:
|
coverage:
|
||||||
weight: 25
|
weight: 15
|
||||||
description: "All six sections are present; STRENGTHS and FORBIDDEN sections are populated, not left empty"
|
description: "All eight sections present and populated. STRENGTHS section includes evidence. FORBIDDEN section is not left empty."
|
||||||
|
|
||||||
|
|||||||
@@ -49,11 +49,16 @@ steps:
|
|||||||
Read the chapter carefully from your editorial perspective.
|
Read the chapter carefully from your editorial perspective.
|
||||||
React to what other editors have written in prior rounds above.
|
React to what other editors have written in prior rounds above.
|
||||||
Where you agree, say so clearly. Where you disagree, argue your position
|
Where you agree, say so clearly. Where you disagree, argue your position
|
||||||
with evidence from the text. If you have changed your mind, say so.
|
with QUOTED EVIDENCE from the text -- do not make claims about the prose
|
||||||
|
without quoting the specific passage you are referring to.
|
||||||
|
If you have changed your mind, say so and quote what changed it.
|
||||||
|
|
||||||
Cover the areas most relevant to your role:
|
Cover the areas most relevant to your role:
|
||||||
- Story structure and pacing
|
- Story structure and pacing
|
||||||
- Character voice and consistency
|
- Character voice and consistency -- if character_profiles is populated below,
|
||||||
|
check each speaking character against their voice constraints and quote any
|
||||||
|
violations you find
|
||||||
|
{character_profiles}
|
||||||
- Line-level clarity and prose quality
|
- Line-level clarity and prose quality
|
||||||
- Continuity with prior chapters and world rules
|
- Continuity with prior chapters and world rules
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user