fix(advice): the pre-token guard, and a stripper that cannot empty real advice
Two findings from the fifth review.
The pre-token fallback re-opened the hole
-----------------------------------------
A reading that has never been edited has no token. The guard fell back
to the gravity check for those, and a comment-only correction leaves the
gravity unchanged - so a late thread's write matched and overwrote the
fresh advice, for exactly the 47 existing readings.
Reproduced live before the fix:
pre-token reading token: None -> stale thread carries ""
fresh write modified: 1
stale write modified: 1 <<< stale advice overwrote the fresh one
final advice: 'STALE ADVICE'
The guard now matches on the ABSENCE of a token, which still tells the
generations apart: an edit writes a token, so a stale thread's "" stops
matching the moment the reading is corrected.
fresh write modified: 1 (must be 1)
stale write modified: 0 (must be 0)
The stripper could empty real advice
------------------------------------
"Actually the drop is fine.\\n\\nWait a couple of days and check again."
is advice, and judging it paragraph by paragraph emptied it - so the
brewer saw nothing at all, with no failure note.
The test is no longer a marker list, which real advice often does not
contain. A reply is all-working when it is LONG and has no instruction
in it: the model's working runs to thousands of characters, while the
advice is two to five sentences.
short reply with no marker -> kept
long reply, no instruction -> dropped
long reply with advice -> advice kept
single-paragraph working -> dropped
Also: the log message named the gravity, but the guard is the token.
182 tests (up from 178).
This commit is contained in:
@@ -276,7 +276,10 @@ def _strip_reasoning(text: str) -> str:
|
||||
# be a no-op, and the brewer would be shown the model thinking about
|
||||
# their beer instead of any advice at all.
|
||||
paragraphs = [p.strip() for p in text.split("\n\n") if p.strip()]
|
||||
if len(paragraphs) > 1 and not any(_looks_like_advice(p) for p in paragraphs):
|
||||
if paragraphs and _is_all_working(text, paragraphs):
|
||||
# It is all working. It is dropped, so the brewer sees a clear
|
||||
# "could not be generated" rather than the model thinking about
|
||||
# their beer.
|
||||
return ""
|
||||
|
||||
for marker in _PREAMBLE_MARKERS:
|
||||
@@ -291,6 +294,9 @@ def _strip_reasoning(text: str) -> str:
|
||||
# running text stops looking like working.
|
||||
paragraphs = [p.strip() for p in tail.split("\n\n") if p.strip()]
|
||||
if len(paragraphs) < 2:
|
||||
# A single paragraph: the whole tail is one block of working
|
||||
# with the advice inside it, so it is kept whole rather than
|
||||
# thrown away.
|
||||
continue
|
||||
|
||||
# Walk back from the end while the paragraph still reads like the
|
||||
@@ -342,6 +348,51 @@ _ADVICE_MARKERS = (
|
||||
)
|
||||
|
||||
|
||||
#: A reply longer than this is the model's working, not the advice.
|
||||
#: The advice is meant to be two to five sentences.
|
||||
_WORKING_LENGTH = 600
|
||||
|
||||
|
||||
def _is_all_working(text: str, paragraphs: list[str]) -> bool:
|
||||
"""True when the reply is the model's working rather than advice.
|
||||
|
||||
Two signals, because either alone is unreliable:
|
||||
|
||||
- **Length.** The working runs to thousands of characters; the
|
||||
advice is two to five sentences.
|
||||
- **Shape.** The working interprets the numbers ("perhaps the yeast
|
||||
is stressed, it could be that..."); the advice tells the brewer
|
||||
what to do.
|
||||
|
||||
A reply that is SHORT is kept even if it has no marker word - real
|
||||
advice is often phrased without one ("Actually the drop is fine.
|
||||
Wait a couple of days and check again.").
|
||||
"""
|
||||
if len(text) <= _WORKING_LENGTH:
|
||||
# Too short to be the model's working. It is advice, whatever
|
||||
# it is worded like.
|
||||
return False
|
||||
|
||||
if _has_advice_marker(text):
|
||||
# It contains an instruction, so part of it is advice.
|
||||
return False
|
||||
|
||||
# Long, and nothing in it reads like an instruction.
|
||||
return True
|
||||
|
||||
|
||||
def _has_advice_marker(text: str) -> bool:
|
||||
"""True when the text contains any advice marker at all.
|
||||
|
||||
Used on the WHOLE reply rather than paragraph by paragraph, because
|
||||
advice can be phrased without any of the markers - "Actually the
|
||||
drop is fine.\n\nWait a couple of days and check again." is advice,
|
||||
and judging it paragraph by paragraph emptied it.
|
||||
"""
|
||||
lowered = text.lower()
|
||||
return any(marker in lowered for marker in _ADVICE_MARKERS)
|
||||
|
||||
|
||||
def _looks_like_advice(paragraph: str) -> bool:
|
||||
"""True when a paragraph reads like advice to the brewer.
|
||||
|
||||
|
||||
@@ -267,14 +267,20 @@ class BeerRepository:
|
||||
# The editing token is the PRIMARY guard: it changes on every
|
||||
# edit, so it catches a stale generation whatever else
|
||||
# changed - the gravity, the comment, or nothing.
|
||||
#
|
||||
# An EMPTY token is not a usable guard: it means the reading
|
||||
# predates the token, so there is no token to match.
|
||||
# Querying on it would match nothing, and the advice would
|
||||
# never be written for the existing readings.
|
||||
query["advice_token"] = str(expected_token)
|
||||
elif expected_token == "":
|
||||
# The reading has NEVER been edited, so it has no token at
|
||||
# all. Match on the ABSENCE of one, which still tells the
|
||||
# generations apart: an edit writes a token, so a stale
|
||||
# thread's "" stops matching the moment the reading is
|
||||
# corrected.
|
||||
#
|
||||
# Falling back to the gravity here would re-open the hole
|
||||
# this guard exists to close - a comment-only correction
|
||||
# leaves the gravity unchanged.
|
||||
query["advice_token"] = {"$exists": False}
|
||||
elif expected_sg is not None:
|
||||
# No token - a direct call with a gravity to check against.
|
||||
# A direct call with a gravity to check against.
|
||||
query["sg"] = str(expected_sg)
|
||||
|
||||
# `$in` matches either form. A plain list would be an exact
|
||||
|
||||
@@ -145,7 +145,8 @@ def _start_advice(run_id: str, reading_id: str, corrected: bool = False) -> None
|
||||
# reading will land in its own turn.
|
||||
log.info(
|
||||
"the advice for reading %s was not stored: the reading "
|
||||
"no longer holds the gravity it was written for",
|
||||
"has been edited since the advice was generated, so the "
|
||||
"advice no longer matches it",
|
||||
read_id,
|
||||
)
|
||||
except Exception: # noqa: BLE001 - never let the thread die noisily
|
||||
|
||||
Reference in New Issue
Block a user