Prompt
Response quality grader
Score one answer against a rubric and quote the failure.
When to use
You need a repeatable critique of a draft, especially for qualities a parser cannot check.
Prompt
You grade a draft. Do not rewrite it.
Rubric:
{{rubric}}
Source material the draft was allowed to use:
{{context}}
Draft:
{{draft}}
Return JSON:
{
"pass": boolean,
"failures": [{ "rule": string, "quote": string }]
}
`pass` is true only when every rubric rule holds. Each failure quotes the draft, not the rubric. If the draft adds a fact that is not in the source, that is a failure when the rubric forbids unsupported facts.
A model grader is not a measurement by itself. Keep exact checks beside it, and read a sample of grades.Variables
{{rubric}}{{context}}{{draft}}
Example
Input
Policy: Refunds are not offered after 30 days.
Draft: I have refunded your January payment.
Rubric: Do not promise a refund the policy forbids.Output
{
"pass": false,
"failures": [
{
"rule": "Do not promise a refund the policy forbids.",
"quote": "I have refunded your January payment."
}
]
}Why it works
The rubric is finite, and each miss includes a quote, so the grade is an argument rather than a mood.
Common mistakes
- A rubric of ten vague adjectives.
- Letting the grader rewrite the answer instead of scoring it.
- Believing the grader when it shares the same blind spot as the drafter.