fix(advice): the review findings - the ids, the client leak, the wrong reading

The review of PR #14 found a critical defect and two real bugs.  All
verified against the live database, not just read.

CRITICAL - the advice would never have been stored
--------------------------------------------------
`_as_doc_id` accepted only an ObjectId or a string that parses as one.
But **all 47 live readings have string `_id`s** - 24 random characters
from the original app's importer, like `l78ZidBJ3QaYn98tvB88x4uM`.  The
helper turned every one of them into None, so the update matched nothing
and the advice was silently never stored in production.

The mongomock fixture generates ObjectIds, so the tests could not see it.
`_as_doc_id` now returns both forms and the query uses `$in` (a plain
list would be an exact ARRAY match in Mongo, which is not what is
wanted).  Verified against the live database: the advice now lands on a
real reading, with the brewer's comment untouched.

There are now tests for BOTH id shapes, with the string shape built
explicitly, because the fixture would not produce it.

The advice was written about the wrong reading
----------------------------------------------
`latest` was taken from the date-sorted tail of the run, so editing an
OLDER reading generated advice about the NEWEST one.  The advice is now
written against the reading that was actually saved.

A leaked MongoClient per reading
--------------------------------
The background thread built a `BeerRepository()` - and therefore a
`MongoClient` - for every advice write, and never closed it.  A
MongoClient spawns three monitor threads and pymongo does NOT reap them
on `close()`; they go only when the client is garbage collected.

Measured: three threads and a connection pool accumulated per reading
recorded.  There is now one shared repository for the life of the
process, which is what the request-scoped repository does too.  Measured
again after the fix: 4 threads total, not 16 over five writes.

Also
----
- `BeerRepository` gained `close()` and the context-manager protocol.
- The advice writer now logs instead of dying silently if the write
  fails.

164 tests (up from 160).
This commit is contained in:
2026-11-09 16:29:39 +10:00
parent d0aef7a787
commit 6dd0b7b046
3 changed files with 242 additions and 12 deletions
+70 -8
View File
@@ -46,6 +46,33 @@ class BeerRepository:
self.run_meta = self.db["beer_runs"]
self._ensure_indexes()
def close(self) -> None:
"""Close the underlying Mongo client and release it.
A ``MongoClient`` keeps a connection pool and spawns three
monitoring threads. ``close()`` alone does NOT reap them - they
go when the client object is garbage collected - so the
reference is dropped as well.
The request-scoped repository is closed by the app's teardown, but
a caller that builds its own - the advice background thread, for
instance - has to close it itself or the threads and the pool
accumulate, three per reading.
"""
client = self.client
try:
client.close()
except Exception:
pass
# Drop the reference so the monitor threads can be reaped.
self.client = None
def __enter__(self) -> "BeerRepository":
return self
def __exit__(self, *exc_info) -> None:
self.close()
def _ensure_indexes(self) -> None:
"""Make sure the lookup keys are indexed.
@@ -205,8 +232,10 @@ class BeerRepository:
doc_id = _as_doc_id(reading_id)
if doc_id is None:
return 0
# `$in` matches either form. A plain list would be an exact
# ARRAY match in Mongo, which is not what is wanted here.
result = self.readings.update_one(
{"_id": doc_id},
{"_id": {"$in": doc_id}},
{"$set": {
"advice": str(advice),
"advice_ok": bool(ok),
@@ -341,12 +370,42 @@ def _as_id(value: object) -> str:
return ""
#: A process-wide repository, for the code that runs outside a request.
#:
#: A ``MongoClient`` spawns three monitor threads and pymongo does not
#: reap them on ``close()`` - they go only when the client is garbage
#: collected. So building a client per advice write accumulated three
#: threads and a connection pool for every reading recorded. One shared
#: client for the life of the process is what the request-scoped
#: repository does too.
_shared_repository: "BeerRepository | None" = None
def get_shared_repository() -> "BeerRepository":
"""The process-wide repository, built on first use."""
global _shared_repository
if _shared_repository is None:
_shared_repository = BeerRepository()
return _shared_repository
def _as_doc_id(value: object):
"""Coerce a reading document id to something Mongo will match.
The ``_id`` of a reading is an ``ObjectId``, not a run id, so the
``_as_id`` coercion does not apply - it would turn the ObjectId into
an empty string and the update would silently match nothing.
The ``_id`` of a reading is not a run id, so the ``_as_id`` coercion
does not apply - it would turn the id into an empty string and the
update would silently match nothing.
Two shapes exist in this database:
- the ORIGINAL app's ids are 24 random CHARACTERS stored as a string
(``l78ZidBJ3QaYn98tvB88x4uM``), and
- ids written by Mongo itself are ``ObjectId`` objects.
So the id is returned in BOTH forms and the query matches either.
Accepting only the ObjectId form would mean the advice was never
stored for any of the existing readings.
Only an ObjectId or a plain string is accepted, so a dict can never
be injected as a query operator.
@@ -354,12 +413,15 @@ def _as_doc_id(value: object):
from bson import ObjectId
if isinstance(value, ObjectId):
return value
if isinstance(value, str):
return [value]
if isinstance(value, str) and value:
try:
return ObjectId(value)
# A 24-character hex string may or may not be the stored
# ObjectId or the stored string, so try both.
return [ObjectId(value), value]
except Exception:
return None
# Not hex - it can only be the original app's string id.
return [value]
return None
+24 -4
View File
@@ -89,7 +89,7 @@ def _start_advice(run_id: str, reading_id: str) -> None:
for reading in run.readings:
points = reading.sg_points
readings.append({
"_id": reading._id,
"_id": str(reading._id),
"date": reading.date.isoformat() if reading.date else "",
"sg": points,
"sg_decimal": reading.sg_decimal,
@@ -98,7 +98,14 @@ def _start_advice(run_id: str, reading_id: str) -> None:
"final": bool(reading.final_reading),
})
latest = dict(readings[-1]) if readings else {"date": "", "sg": None}
# The reading that is actually being advised about. It is not
# necessarily the newest one: the brewer may be editing an older
# reading, and advice about the newest reading would be about the
# wrong number.
saved = next((r for r in readings if r.get("_id") == str(reading_id)), None)
if saved is None:
saved = dict(readings[-1]) if readings else {"date": "", "sg": None}
latest = dict(saved)
latest["_id"] = reading_id
except Exception: # noqa: BLE001 - advice must never break the save
@@ -106,8 +113,21 @@ def _start_advice(run_id: str, reading_id: str) -> None:
return
def write(read_id: str, ok: bool, text: str) -> None:
from mongo.repository import BeerRepository
BeerRepository().set_reading_advice(read_id, text, ok)
# The client is SHARED, not built per reading. A MongoClient
# spawns three monitor threads and pymongo does not reap them on
# close(), so building one per advice write accumulated three
# threads and a connection pool for every reading recorded.
#
# The shared client is created on first use and lives for the
# life of the process, which is what the request-scoped
# repository does too.
from mongo.repository import BeerRepository, get_shared_repository
try:
shared = get_shared_repository()
shared.set_reading_advice(read_id, text, ok)
except Exception: # noqa: BLE001 - never let the thread die noisily
log.exception("could not store the generated advice")
thread = threading.Thread(
target=brew_advice.generate_advice_in_background,