armistace/beer_data
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(advice): a deterministic split, not a guess at where the advice starts

Browse Source 
The stripper tried to work out where the model's working ends and the
advice begins by classifying prose - marker words, hedging words,
length.  It failed in BOTH directions, and the review caught both:

- it emptied legitimate advice (a 1,093-character reply containing one
  hedging word and no marker phrase was reduced to ""), so the brewer
  saw nothing at all; and
- it let the model's working through in full (a 1,053-character reply
  with no hedging word was returned verbatim), so the brewer was shown
  the model thinking about their beer.

Neither a marker list nor a length threshold is reliable.  Prose is not
a format.

So the model is asked for the format instead.  Its reply must have this
shape:

    ADVICE:
    <the advice itself>

Everything before the marker is working, everything from it on is the
advice.  The split is exact, with no judgement calls.

Verified against the live model, four different brews:

    Test Pale Ale        marker@pos=0  finish=stop  advice_len=656
    Nocturna Christmas   marker@pos=0  finish=stop  advice_len=668
    Ginger beer          marker@pos=0  finish=stop  advice_len=621
    Cider raspberry      marker@pos=0  finish=stop  advice_len=539

All four start clean, with no marker left in the advice.

A reply with no marker is used as it stands - the safe direction, where
the brewer sees a slightly rambling reply rather than no advice at all.

An empty advice is now recorded as a FAILURE rather than stored as a
successful generation.  Storing an empty string with advice_ok=True
made the page render "Thinking..." for ever, because the template's
`{% if reading.advice %}` was false - so the brewer never learned that
it had failed.

181 tests (up from 184 - the eight heuristic-era stripper tests are
replaced by six that test the marker behaviour).
This commit is contained in:
hermes
2026-11-09 19:44:52 +10:00
parent cbc71716e8
commit 75f94d47e2
2 changed files with 99 additions and 313 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/flask/brew_advice.py
+44 -188
View File
@@ -78,6 +78,13 @@ temperature?
- Should they rack, add more hops, raise the temperature, or wait?
Rules:
- Your reply must have this exact shape:
ADVICE:
<the advice itself, two to five sentences of plain prose>
The line "ADVICE:" comes FIRST, on its own line, with nothing before
it. Do not explain your reasoning and do not show your working.
- Advise about the reading that is marked as the one to advise on. It
may not be the most recent one - the brewer may be correcting an older
reading - and the advice must be about the reading they asked about.
@@ -213,9 +220,11 @@ def call_ollama(prompt: str, *, url: str | None = None, model: str | None = None
truncated = finish_reason == "length"
advice = _trim_advice(_strip_reasoning(content))
if truncated:
advice = _trim_advice(advice + " (the reply was cut off)")
return advice
return advice, truncated
def _trim_advice(text: str) -> str:
@@ -239,199 +248,35 @@ def _trim_advice(text: str) -> str:
return tail.strip()
#: The reasoning models sometimes put their working IN the content, even
#: with thinking turned off - "Let me look at this..." for several
#: thousand characters before the actual advice. The advice is the last
#: few paragraphs, so the preamble is cut off.
_PREAMBLE_MARKERS = (
"let me look",
"let me think",
"let me work",
"okay, let me",
"first, let me",
"hmm,",
"wait,",
"let me analyze",
"let me consider",
)
#: The marker the model is asked to put at the start of its advice.
#:
#: Guessing where the advice begins by classifying prose is unreliable
#: in both directions - it emptied legitimate advice, and it let the
#: model's working through in full. A marker the model writes itself is
#: deterministic: everything before it is working, everything from it on
#: is the advice.
_ADVICE_MARKER = "ADVICE:"
def _strip_reasoning(text: str) -> str:
"""Cut a reasoning preamble off the front of a reply.
"""Take the advice out of a reply.
The advice is the prose at the END - the working comes first, in
several paragraphs. So the preamble is removed by finding the last
paragraph break before the advice starts, not the first.
The model is asked to put ``ADVICE:`` on its own line at the start
of the advice, so the working before it can be dropped exactly, and
the advice kept exactly.
Only a preamble that looks like working is removed; a reply that
starts straight into the advice is left alone.
A reply with no marker is used as it stands. That is the safe
direction: the model was asked for the marker, so a reply without
one is one where it wrote the advice straight away - and if the
working did leak in, the brewer sees a slightly rambling reply
rather than no advice at all.
"""
lowered = text.lower()
index = text.find(_ADVICE_MARKER)
if index == -1:
return text.strip()
# A reply that is ALL working - no advice paragraph at all - has to
# be caught too. The marker phrases are a hint, not the only signal:
# a reply can be pure working without ever saying "let me think".
#
# The working is DROPPED, not returned. Returning it unchanged would
# be a no-op, and the brewer would be shown the model thinking about
# their beer instead of any advice at all.
paragraphs = [p.strip() for p in text.split("\n\n") if p.strip()]
if paragraphs and _is_all_working(text, paragraphs):
# It is all working. It is dropped, so the brewer sees a clear
# "could not be generated" rather than the model thinking about
# their beer.
return ""
for marker in _PREAMBLE_MARKERS:
index = lowered.find(marker)
# index 0 is fine - a reply can START with "Let me look at this".
if index < 0:
continue
tail = text[index:]
# The advice is the LAST paragraph of the tail. Split on the
# blank lines and take the paragraphs from the end, until the
# running text stops looking like working.
paragraphs = [p.strip() for p in tail.split("\n\n") if p.strip()]
if len(paragraphs) < 2:
# A single paragraph: the whole tail is one block of working
# with the advice inside it, so it is kept whole rather than
# thrown away.
continue
# Walk back from the end while the paragraph still reads like the
# model's working rather than the brewer's advice.
kept = [paragraphs[-1]]
for paragraph in reversed(paragraphs[:-1]):
if _looks_like_advice(paragraph):
kept.append(paragraph)
else:
break
kept.reverse()
candidate = "\n\n".join(kept).strip()
if len(candidate) >= 80:
return candidate
return text
#: Words that mark a paragraph as the ADVICE rather than the model's
#: working. The advice talks about what the brewer should DO.
#: Phrases that mark a paragraph as the ADVICE rather than the model's
#: working. They are matched as PHRASES, not bare words - "keep" alone
#: matches "I need to keep in mind that the hydrometer was read at a
#: different temperature", which is working, not advice.
_ADVICE_MARKERS = (
"you should",
"you could",
"you may want",
"you might",
"you will want",
"give it",
"let it sit",
"let it ferment",
"take another reading",
"take a reading",
"raise it to",
"raise the",
"lower the",
"rack it",
"you could repitch",
"i would suggest",
"i suggest",
"i recommend",
"in your case",
"here is",
"here's",
"don't",
"do not",
)
#: A reply longer than this is the model's working, not the advice.
#: The advice is meant to be two to five sentences.
_WORKING_LENGTH = 600
def _is_all_working(text: str, paragraphs: list[str]) -> bool:
"""True when the reply is the model's working rather than advice.
The reliable signal is SHAPE, not length. A legitimate advice reply
can be a thousand characters, and a short working reply can be a
few hundred - so length alone gets both wrong.
The model's working is about INTERPRETING the numbers: what the
reading might mean, what could be causing it, what it is unsure
about. The advice is about what to DO.
So a reply is working when it is full of the model's hedging about
the data AND contains no instruction at all.
"""
if _has_advice_marker(text):
# It contains an instruction, so part of it is advice.
return False
lowered = text.lower()
hedging = sum(lowered.count(word) for word in _HEDGING_WORDS)
if hedging == 0:
# No instruction and no hedging about the data. This is advice,
# however long it is.
return False
if len(text) <= _WORKING_LENGTH:
# Some hedging, but it is short - that is a brewer's note that
# happens to use a hedging word, not a page of working.
return False
# Hedging, no instruction, and long: the working.
return True
#: The words the model uses when it is working through the data rather
#: than telling the brewer what to do.
_HEDGING_WORDS = (
"perhaps", "maybe", "possibly", "could be", "might be", "might",
"i think", "i am not sure", "i'm not sure", "actually", "hmm",
"wait,", "let me", "i need to", "the question is", "unclear",
"i am unsure", "not sure", "unfortunately",
)
def _has_advice_marker(text: str) -> bool:
"""True when the text contains any advice marker at all.
Used on the WHOLE reply rather than paragraph by paragraph, because
advice can be phrased without any of the markers - "Actually the
drop is fine.\n\nWait a couple of days and check again." is advice,
and judging it paragraph by paragraph emptied it.
"""
lowered = text.lower()
return any(marker in lowered for marker in _ADVICE_MARKERS)
def _looks_like_advice(paragraph: str) -> bool:
"""True when a paragraph reads like advice to the brewer.
The model's working is full of hedging - "perhaps", "or maybe", "it
could be" - and is about interpreting the numbers. The advice is
about what to do.
"""
lowered = paragraph.lower()
if any(marker in lowered for marker in _ADVICE_MARKERS):
return True
# A paragraph that reads like working is not advice. The signs are
# hedging ("perhaps", "could be"), and the model talking to itself
# ("wait", "hmm", "so", "actually") rather than to the brewer.
working_words = ("perhaps", "maybe", "possibly", "could be", "might be",
"i think", "i am not sure", "actually", "hmm", "wait",
"let me", "so on brew day", "there is a discrepancy",
"that is a discrepancy", "which means",
"i need to keep in mind", "the question is",
"this could mean", "it could be", "or maybe")
if any(word in lowered for word in working_words):
return False
return True
advice = text[index + len(_ADVICE_MARKER):].strip()
return advice or ""
def generate_advice(run: dict, reading: dict, history: list[dict], *,
@@ -446,7 +291,18 @@ def generate_advice(run: dict, reading: dict, history: list[dict], *,
"""
prompt = build_prompt(run, reading, history, corrected=corrected)
try:
return True, call_ollama(prompt, url=url, model=model, timeout=timeout)
advice, _truncated = call_ollama(prompt, url=url, model=model,
timeout=timeout)
if not advice:
# The reply had no advice in it - the model wrote only its
# working, or the marker was missing and nothing followed it.
# Storing an empty string with advice_ok=True would render
# as "Thinking..." for ever, so it is recorded as a failure
# with a note the brewer can see.
log.warning("the model returned no advice")
return False, ("Advice could not be generated - the model "
"returned only its reasoning.")
return True, advice
except urllib.error.HTTPError as exc:
# HTTPError is a subclass of URLError, so it has to be caught
# FIRST or it would be swallowed by the unreachable branch.