fix(advice): the review findings - the ids, the client leak, the wrong reading
The review of PR #14 found a critical defect and two real bugs. All verified against the live database, not just read. CRITICAL - the advice would never have been stored -------------------------------------------------- `_as_doc_id` accepted only an ObjectId or a string that parses as one. But **all 47 live readings have string `_id`s** - 24 random characters from the original app's importer, like `l78ZidBJ3QaYn98tvB88x4uM`. The helper turned every one of them into None, so the update matched nothing and the advice was silently never stored in production. The mongomock fixture generates ObjectIds, so the tests could not see it. `_as_doc_id` now returns both forms and the query uses `$in` (a plain list would be an exact ARRAY match in Mongo, which is not what is wanted). Verified against the live database: the advice now lands on a real reading, with the brewer's comment untouched. There are now tests for BOTH id shapes, with the string shape built explicitly, because the fixture would not produce it. The advice was written about the wrong reading ---------------------------------------------- `latest` was taken from the date-sorted tail of the run, so editing an OLDER reading generated advice about the NEWEST one. The advice is now written against the reading that was actually saved. A leaked MongoClient per reading -------------------------------- The background thread built a `BeerRepository()` - and therefore a `MongoClient` - for every advice write, and never closed it. A MongoClient spawns three monitor threads and pymongo does NOT reap them on `close()`; they go only when the client is garbage collected. Measured: three threads and a connection pool accumulated per reading recorded. There is now one shared repository for the life of the process, which is what the request-scoped repository does too. Measured again after the fix: 4 threads total, not 16 over five writes. Also ---- - `BeerRepository` gained `close()` and the context-manager protocol. - The advice writer now logs instead of dying silently if the write fails. 164 tests (up from 160).
This commit is contained in:
@@ -46,6 +46,33 @@ class BeerRepository:
|
||||
self.run_meta = self.db["beer_runs"]
|
||||
self._ensure_indexes()
|
||||
|
||||
def close(self) -> None:
|
||||
"""Close the underlying Mongo client and release it.
|
||||
|
||||
A ``MongoClient`` keeps a connection pool and spawns three
|
||||
monitoring threads. ``close()`` alone does NOT reap them - they
|
||||
go when the client object is garbage collected - so the
|
||||
reference is dropped as well.
|
||||
|
||||
The request-scoped repository is closed by the app's teardown, but
|
||||
a caller that builds its own - the advice background thread, for
|
||||
instance - has to close it itself or the threads and the pool
|
||||
accumulate, three per reading.
|
||||
"""
|
||||
client = self.client
|
||||
try:
|
||||
client.close()
|
||||
except Exception:
|
||||
pass
|
||||
# Drop the reference so the monitor threads can be reaped.
|
||||
self.client = None
|
||||
|
||||
def __enter__(self) -> "BeerRepository":
|
||||
return self
|
||||
|
||||
def __exit__(self, *exc_info) -> None:
|
||||
self.close()
|
||||
|
||||
def _ensure_indexes(self) -> None:
|
||||
"""Make sure the lookup keys are indexed.
|
||||
|
||||
@@ -205,8 +232,10 @@ class BeerRepository:
|
||||
doc_id = _as_doc_id(reading_id)
|
||||
if doc_id is None:
|
||||
return 0
|
||||
# `$in` matches either form. A plain list would be an exact
|
||||
# ARRAY match in Mongo, which is not what is wanted here.
|
||||
result = self.readings.update_one(
|
||||
{"_id": doc_id},
|
||||
{"_id": {"$in": doc_id}},
|
||||
{"$set": {
|
||||
"advice": str(advice),
|
||||
"advice_ok": bool(ok),
|
||||
@@ -341,12 +370,42 @@ def _as_id(value: object) -> str:
|
||||
return ""
|
||||
|
||||
|
||||
#: A process-wide repository, for the code that runs outside a request.
|
||||
#:
|
||||
#: A ``MongoClient`` spawns three monitor threads and pymongo does not
|
||||
#: reap them on ``close()`` - they go only when the client is garbage
|
||||
#: collected. So building a client per advice write accumulated three
|
||||
#: threads and a connection pool for every reading recorded. One shared
|
||||
#: client for the life of the process is what the request-scoped
|
||||
#: repository does too.
|
||||
_shared_repository: "BeerRepository | None" = None
|
||||
|
||||
|
||||
def get_shared_repository() -> "BeerRepository":
|
||||
"""The process-wide repository, built on first use."""
|
||||
global _shared_repository
|
||||
|
||||
if _shared_repository is None:
|
||||
_shared_repository = BeerRepository()
|
||||
return _shared_repository
|
||||
|
||||
|
||||
def _as_doc_id(value: object):
|
||||
"""Coerce a reading document id to something Mongo will match.
|
||||
|
||||
The ``_id`` of a reading is an ``ObjectId``, not a run id, so the
|
||||
``_as_id`` coercion does not apply - it would turn the ObjectId into
|
||||
an empty string and the update would silently match nothing.
|
||||
The ``_id`` of a reading is not a run id, so the ``_as_id`` coercion
|
||||
does not apply - it would turn the id into an empty string and the
|
||||
update would silently match nothing.
|
||||
|
||||
Two shapes exist in this database:
|
||||
|
||||
- the ORIGINAL app's ids are 24 random CHARACTERS stored as a string
|
||||
(``l78ZidBJ3QaYn98tvB88x4uM``), and
|
||||
- ids written by Mongo itself are ``ObjectId`` objects.
|
||||
|
||||
So the id is returned in BOTH forms and the query matches either.
|
||||
Accepting only the ObjectId form would mean the advice was never
|
||||
stored for any of the existing readings.
|
||||
|
||||
Only an ObjectId or a plain string is accepted, so a dict can never
|
||||
be injected as a query operator.
|
||||
@@ -354,12 +413,15 @@ def _as_doc_id(value: object):
|
||||
from bson import ObjectId
|
||||
|
||||
if isinstance(value, ObjectId):
|
||||
return value
|
||||
if isinstance(value, str):
|
||||
return [value]
|
||||
if isinstance(value, str) and value:
|
||||
try:
|
||||
return ObjectId(value)
|
||||
# A 24-character hex string may or may not be the stored
|
||||
# ObjectId or the stored string, so try both.
|
||||
return [ObjectId(value), value]
|
||||
except Exception:
|
||||
return None
|
||||
# Not hex - it can only be the original app's string id.
|
||||
return [value]
|
||||
return None
|
||||
|
||||
|
||||
|
||||
@@ -89,7 +89,7 @@ def _start_advice(run_id: str, reading_id: str) -> None:
|
||||
for reading in run.readings:
|
||||
points = reading.sg_points
|
||||
readings.append({
|
||||
"_id": reading._id,
|
||||
"_id": str(reading._id),
|
||||
"date": reading.date.isoformat() if reading.date else "",
|
||||
"sg": points,
|
||||
"sg_decimal": reading.sg_decimal,
|
||||
@@ -98,7 +98,14 @@ def _start_advice(run_id: str, reading_id: str) -> None:
|
||||
"final": bool(reading.final_reading),
|
||||
})
|
||||
|
||||
latest = dict(readings[-1]) if readings else {"date": "", "sg": None}
|
||||
# The reading that is actually being advised about. It is not
|
||||
# necessarily the newest one: the brewer may be editing an older
|
||||
# reading, and advice about the newest reading would be about the
|
||||
# wrong number.
|
||||
saved = next((r for r in readings if r.get("_id") == str(reading_id)), None)
|
||||
if saved is None:
|
||||
saved = dict(readings[-1]) if readings else {"date": "", "sg": None}
|
||||
latest = dict(saved)
|
||||
latest["_id"] = reading_id
|
||||
|
||||
except Exception: # noqa: BLE001 - advice must never break the save
|
||||
@@ -106,8 +113,21 @@ def _start_advice(run_id: str, reading_id: str) -> None:
|
||||
return
|
||||
|
||||
def write(read_id: str, ok: bool, text: str) -> None:
|
||||
from mongo.repository import BeerRepository
|
||||
BeerRepository().set_reading_advice(read_id, text, ok)
|
||||
# The client is SHARED, not built per reading. A MongoClient
|
||||
# spawns three monitor threads and pymongo does not reap them on
|
||||
# close(), so building one per advice write accumulated three
|
||||
# threads and a connection pool for every reading recorded.
|
||||
#
|
||||
# The shared client is created on first use and lives for the
|
||||
# life of the process, which is what the request-scoped
|
||||
# repository does too.
|
||||
from mongo.repository import BeerRepository, get_shared_repository
|
||||
|
||||
try:
|
||||
shared = get_shared_repository()
|
||||
shared.set_reading_advice(read_id, text, ok)
|
||||
except Exception: # noqa: BLE001 - never let the thread die noisily
|
||||
log.exception("could not store the generated advice")
|
||||
|
||||
thread = threading.Thread(
|
||||
target=brew_advice.generate_advice_in_background,
|
||||
|
||||
Reference in New Issue
Block a user