fix(advice): judge the working by its shape, not by its length

A length threshold is a blunt instrument: a legitimate advice reply can
run to a thousand characters, and a short working reply can be a few
hundred - so a threshold gets both wrong.  A 995-character advice reply
with no marker word in it was being emptied.

The reliable signal is the SHAPE.  The model's working is about
INTERPRETING the numbers - what the reading might mean, what could be
causing it, what it is unsure about.  The advice is about what to DO.

So a reply is working when it is full of the model's hedging about the
data AND contains no instruction at all.  Length is only a tie-breaker
for a reply that is both hedged and short.

    995-char legitimate advice  -> kept
    long pure working           -> dropped
    short legitimate advice     -> kept
    long working + advice       -> advice kept

Checked against the live model - a real reply comes back clean, with no
reasoning in it:

    At two weeks in, dropping from 1.055 to 1.020 is a healthy 35-point
    fall - about 64% attenuation - which is right in the normal range for
    a pale ale, though you likely still have 5 to 8 points to go...

184 tests (up from 183).
This commit is contained in:
2026-11-09 19:27:08 +10:00
parent 4bedd23c24
commit cbc71716e8
2 changed files with 61 additions and 15 deletions
+32 -15
View File
@@ -356,31 +356,48 @@ _WORKING_LENGTH = 600
def _is_all_working(text: str, paragraphs: list[str]) -> bool:
"""True when the reply is the model's working rather than advice.
Two signals, because either alone is unreliable:
The reliable signal is SHAPE, not length. A legitimate advice reply
can be a thousand characters, and a short working reply can be a
few hundred - so length alone gets both wrong.
- **Length.** The working runs to thousands of characters; the
advice is two to five sentences.
- **Shape.** The working interprets the numbers ("perhaps the yeast
is stressed, it could be that..."); the advice tells the brewer
what to do.
The model's working is about INTERPRETING the numbers: what the
reading might mean, what could be causing it, what it is unsure
about. The advice is about what to DO.
A reply that is SHORT is kept even if it has no marker word - real
advice is often phrased without one ("Actually the drop is fine.
Wait a couple of days and check again.").
So a reply is working when it is full of the model's hedging about
the data AND contains no instruction at all.
"""
if len(text) <= _WORKING_LENGTH:
# Too short to be the model's working. It is advice, whatever
# it is worded like.
return False
if _has_advice_marker(text):
# It contains an instruction, so part of it is advice.
return False
# Long, and nothing in it reads like an instruction.
lowered = text.lower()
hedging = sum(lowered.count(word) for word in _HEDGING_WORDS)
if hedging == 0:
# No instruction and no hedging about the data. This is advice,
# however long it is.
return False
if len(text) <= _WORKING_LENGTH:
# Some hedging, but it is short - that is a brewer's note that
# happens to use a hedging word, not a page of working.
return False
# Hedging, no instruction, and long: the working.
return True
#: The words the model uses when it is working through the data rather
#: than telling the brewer what to do.
_HEDGING_WORDS = (
"perhaps", "maybe", "possibly", "could be", "might be", "might",
"i think", "i am not sure", "i'm not sure", "actually", "hmm",
"wait,", "let me", "i need to", "the question is", "unclear",
"i am unsure", "not sure", "unfortunately",
)
def _has_advice_marker(text: str) -> bool:
"""True when the text contains any advice marker at all.