From 9c362b92f82324cb7ba4144a12fb95a6fabe65ee Mon Sep 17 00:00:00 2001 From: David Baity <158404677+MazakTheDwarf@users.noreply.github.com> Date: Wed, 1 Apr 2026 22:33:53 -0400 Subject: [PATCH] Strengthen chapter review quality gates - chapter_review.yml: mandatory PROSE EVIDENCE section (3-5 verbatim quotes) - CHARACTER VOICE AUDIT with per-constraint checks and violation quoting - Score anchors (95-100/85-94/70-84/<70) to break rubber-stamp clustering - MUST-FIX items require ORIGINAL quote + PROBLEM + FIX rewrite - Adjudication: prose_evidence(35) specificity(30) voice_audit(20) coverage(15) - chapter_roundtable.yml: require quoted evidence, add character_profiles check --- templates/chapter_review.yml | 117 ++++++++++++++++++++----------- templates/chapter_roundtable.yml | 9 ++- 2 files changed, 82 insertions(+), 44 deletions(-) diff --git a/templates/chapter_review.yml b/templates/chapter_review.yml index 726fe78..9a0e239 100644 --- a/templates/chapter_review.yml +++ b/templates/chapter_review.yml @@ -1,7 +1,8 @@ name: chapter_review description: > Independent chapter review. Agent reads the chapter and produces a structured - critique in their editorial domain with explicit must-fix vs optional categorization. + critique with mandatory verbatim prose quotes, score-anchored verdict, and + explicit character voice violation checks. debug: true rag_exclude: [business_plan, charter] model: power @@ -18,7 +19,7 @@ sections: steps: - type: think - max_tokens: 4000 + max_tokens: 5000 hint: | TARGET AUDIENCE: {genre_audience} GENRE: {genre_name} @@ -29,54 +30,82 @@ steps: {chapter_text} --- - Write a detailed editorial review from your perspective. - Be specific — cite line numbers or quote passages where relevant. - Structure your review using EXACTLY these six labeled sections: + Write a detailed editorial review. You MUST quote the actual text to support every + claim you make. Vague commentary ("the pacing drags") without a quoted passage is + not acceptable and will cause this review to fail adjudication. - 1. STRENGTHS TO PRESERVE - List elements that are working well and must survive the editorial process unchanged. - Be specific: quote or paraphrase the passage. Vague praise ("the writing is good") - is not useful here. - ALSO CHECK VOICE SIGNATURES: - CHARACTER VOICE PROFILES: + Structure your review using EXACTLY these seven labeled sections: + + 1. PROSE EVIDENCE + Quote 3-5 passages verbatim from this chapter (use quotation marks and indicate + approximate position: "early", "mid", "late"). + For each quote, give one sentence of inline commentary: what it does well or + what it fails to do. This section is MANDATORY -- do not summarize or paraphrase. + A review with zero verbatim quotes will automatically score below 70. + + 2. CHARACTER VOICE AUDIT {character_profiles} - If the above block is non-empty: verify each named character in this chapter - speaks to their profile. Can you identify each character's dialogue without - speaker tags? Note YES or NO for each main character. - If the block is empty: check RAG/project context for "## Voice Signatures" - and note whether voice differentiation is present. + If the above block is non-empty: for EACH named character who speaks in this + chapter, check their voice against their profile. Quote one line of their + dialogue and state YES or NO for each constraint: + - Does the character use their signature vocabulary / verbal tics? + - Do they avoid any explicitly forbidden speech patterns? + - Is their emotional register consistent with their arc position? + If a violation is found, quote the offending line and state exactly what rule + is broken (e.g. "Dorian uses contraction 'don't' -- profile forbids contractions + unless in extreme pain or physical exhaustion. This scene does not qualify."). + If the block is empty: check RAG context for "## Voice Signatures" and report + what you find. - 2. MUST-FIX — CONTINUITY + 3. STRENGTHS TO PRESERVE + List 2-4 elements working well that must survive the editorial process unchanged. + Each item must include a verbatim quote or specific scene reference. + Generic praise ("the atmosphere is strong") without evidence is not allowed here. + + 4. MUST-FIX -- CONTINUITY List factual errors, POV breaks, timeline inconsistencies, or world-rule violations. - These are non-negotiable — they cannot be left as "optional." For each item, state: - - What the error is - - What the correction should be + These are non-negotiable. For each item state: + - ORIGINAL: quote the offending passage verbatim + - PROBLEM: what rule or established fact it violates + - FIX: the concrete correction (rewrite the sentence if necessary) - 3. MUST-FIX — CLARITY + 5. MUST-FIX -- CLARITY List passages where meaning is genuinely obscured, transitions are dropped, or threads are left dangling in a way that blocks reader comprehension. For each item: - - Quote or reference the passage - - State the concrete fix + - ORIGINAL: quote the offending passage verbatim + - PROBLEM: why it blocks comprehension + - FIX: the concrete correction - 4. OPTIONAL SUGGESTIONS + 6. OPTIONAL SUGGESTIONS Craft improvements that would strengthen the chapter but are NOT required for it to - pass. Label each suggestion clearly as optional. Do not inflate this section — only - include suggestions with a clear upside and low risk of voice damage. - ⚠️ Do NOT suggest adding thematic contrast metaphors, making dialogue more elegant, - or smoothing sentence variation — these are voice PRESERVATION issues, not fixes. + pass. Label each suggestion clearly as optional. Include the relevant quote. + Do not inflate this section -- only include suggestions with a clear upside and low + risk of voice damage. + Do NOT suggest adding thematic contrast metaphors, making dialogue more elegant, + or smoothing sentence variation -- these are voice PRESERVATION issues, not fixes. - 5. FORBIDDEN CHANGES / NON-GOALS - List things that might appear to be problems but should NOT be changed — intentional - voice choices, genre conventions, structural decisions that are working. This section - protects the draft from over-editing. - ALWAYS list here: verbal tics, repeated phrases, and "imperfect" speech that is + 7. FORBIDDEN CHANGES / NON-GOALS + List things that might appear to be problems but should NOT be changed -- intentional + voice choices, genre conventions, structural decisions that are working. + ALWAYS include: verbal tics, repeated phrases, and "imperfect" speech that is clearly a character signature, not an error. - 6. VERDICT + 8. VERDICT One of: PASS / REVISE + SCORE ANCHORS -- be honest, scores that cluster at 90-96 without strong evidence + signal a rubber-stamp review and will be rejected by the adjudicator: + 95-100: Zero MUST-FIX items. PROSE EVIDENCE quotes show above-average craft. + Voice audit finds no violations. + 85-94: 1-2 minor MUST-FIX items. At least one PROSE EVIDENCE quote shows + a clear weakness with explanation. + 70-84: Clear structural or voice problems. Every MUST-FIX item is quoted + verbatim with a rewrite suggestion. + Below 70: Multiple serious violations. Every failing passage is quoted and + corrected. Automatic REVISE verdict. + + State your score and a one-sentence justification citing your evidence. Use PASS if only optional improvements remain. Use REVISE if any MUST-FIX items are present. - Reserve judgment on full rewrites — that decision belongs to the roundtable. - type: document filename: "Chapter_{chapter_number}_review_{review_letter}" @@ -89,12 +118,16 @@ adjudication: pass_threshold: 60 deliverable_type: editorial_review criteria: - specificity: - weight: 40 - description: "Review cites specific passages, scenes, or structural elements in all sections — not vague general commentary" - actionability: + prose_evidence: weight: 35 - description: "Each MUST-FIX item includes a concrete, implementable correction; OPTIONAL items are clearly labeled" + description: "Review contains 3+ verbatim quoted passages with inline commentary. Zero quotes = automatic fail. Paraphrases do not count as quotes." + specificity: + weight: 30 + description: "Every MUST-FIX item includes the original passage quoted verbatim AND a concrete rewrite or correction. Generic commentary without quotes scores zero here." + voice_audit: + weight: 20 + description: "CHARACTER VOICE AUDIT section checks each speaking character against their profile constraints. Each violation is quoted and the broken rule named explicitly." coverage: - weight: 25 - description: "All six sections are present; STRENGTHS and FORBIDDEN sections are populated, not left empty" + weight: 15 + description: "All eight sections present and populated. STRENGTHS section includes evidence. FORBIDDEN section is not left empty." + diff --git a/templates/chapter_roundtable.yml b/templates/chapter_roundtable.yml index d6cee03..955426e 100644 --- a/templates/chapter_roundtable.yml +++ b/templates/chapter_roundtable.yml @@ -49,11 +49,16 @@ steps: Read the chapter carefully from your editorial perspective. React to what other editors have written in prior rounds above. Where you agree, say so clearly. Where you disagree, argue your position - with evidence from the text. If you have changed your mind, say so. + with QUOTED EVIDENCE from the text -- do not make claims about the prose + without quoting the specific passage you are referring to. + If you have changed your mind, say so and quote what changed it. Cover the areas most relevant to your role: - Story structure and pacing - - Character voice and consistency + - Character voice and consistency -- if character_profiles is populated below, + check each speaking character against their voice constraints and quote any + violations you find + {character_profiles} - Line-level clarity and prose quality - Continuity with prior chapters and world rules