fix(advice): stale advice must not overwrite a correction
The review found a race between the background advice thread and the
brewer correcting a reading.
`set_reading_advice` wrote unconditionally by id. The advice is
generated in the background, so if the brewer corrects a reading while
the advice for the OLD gravity is still being generated, the slow thread
lands afterwards and overwrites the fresh advice with advice about a
reading that no longer exists. The reading ends up with the final
gravity from the correction and the advice from before it.
Reproduced against the live database:
wrote advice for 1050
wrote advice for 1020
late stale write matched: 0
final sg: 1020 | final advice: 'ADVICE FOR THE CORRECTION'
The write is now conditional on the reading still holding the gravity
the advice was generated for, so a late thread matches nothing.
Also
----
`finish_reason` is now checked. A reply that stopped at the token
ceiling is incomplete - it may be advice that stops mid-sentence - and
it was being stored as if it were the whole answer. It is now marked
as cut off.
173 tests (up from 171).
This commit is contained in:
@@ -112,20 +112,25 @@ def _start_advice(run_id: str, reading_id: str, corrected: bool = False) -> None
|
||||
log.exception("could not gather the reading for the advice")
|
||||
return
|
||||
|
||||
# The gravity the advice is about. The write is conditional on the
|
||||
# reading still holding it, so a slow advice thread cannot overwrite
|
||||
# a correction with advice about the old value.
|
||||
advice_sg = latest.get("sg")
|
||||
|
||||
def write(read_id: str, ok: bool, text: str) -> None:
|
||||
# The client is SHARED, not built per reading. A MongoClient
|
||||
# spawns three monitor threads and pymongo does not reap them on
|
||||
# close(), so building one per advice write accumulated three
|
||||
# spawns three monitor threads and pymongo does not reap them
|
||||
# on close(), so building one per advice write accumulated three
|
||||
# threads and a connection pool for every reading recorded.
|
||||
#
|
||||
# The shared client is created on first use and lives for the
|
||||
# life of the process, which is what the request-scoped
|
||||
# repository does too.
|
||||
from mongo.repository import BeerRepository, get_shared_repository
|
||||
from mongo.repository import get_shared_repository
|
||||
|
||||
try:
|
||||
shared = get_shared_repository()
|
||||
shared.set_reading_advice(read_id, text, ok)
|
||||
shared.set_reading_advice(read_id, text, ok, expected_sg=advice_sg)
|
||||
except Exception: # noqa: BLE001 - never let the thread die noisily
|
||||
log.exception("could not store the generated advice")
|
||||
|
||||
|
||||
Reference in New Issue
Block a user