fix(advice): guard the advice write on the comment as well as the token #21

Merged
hermes merged 1 commits from fix/advice-comment-guard into master 2026-11-09 22:27:01 +10:00
Collaborator

A fifth review of the PR, run against the pre-merge branch. One of its findings is a real hole worth closing even though the app cannot currently reach it.

The finding

set_reading_advice guards the write on the editing token. The token is written by upsert_reading on every edit, so every edit the application makes is caught.

But a document edited outside the application - by a script, or by hand in a database tool - keeps its empty token. The reviewer reproduced it against live Mongo:

pre-token reading token: None  ->  stale thread carries ""
fresh write matched: 1
stale write matched: 1   <<< stale advice overwrote the fresh one
final advice: 'STALE ADVICE'

That edit bypassed upsert_reading, so no token was written.

The fix

The advice is generated from the comment as well as the gravity, so the comment is now a second, independent guard on the write. A stale write can only land if both the gravity and the comment are unchanged.

query["comment"] = {
    "$regex": "^" + re.escape(expected_comment.strip()) + r"\s*$"
}

The match is a stripped, anchored regex rather than an exact one, because the stored comment and the one the view sees can differ by trailing whitespace - two of your live readings are padded ('First time bottling ').

Verified against live Mongo

With the real repository code:

stored comment has trailing whitespace: 'First time bottling '
  write with the stripped comment matched: 1   <- the guard is not too strict
  stale write after an outside edit matched: 0 <- correctly refused
  FINAL advice: 'FRESH' | comment: 'edited outside'

And the normal application path is unaffected - a write with the real token and the current comment still lands.

Also in this PR

  • The log message said the reading "no longer holds the gravity it was written for", but the guard is the token and the comment. It now says what actually happened.

The review's other findings

  • LOW - the "all working" branch could empty a legitimate advice reply and LOW - single-paragraph pure working is still stored as advice: both describe _looks_like_advice and _ADVICE_MARKERS, which do not exist in master. The marker-word heuristics were replaced by the deterministic ADVICE: marker split in 75f94d4, and master has neither function.
  • The empty-token hole it was asked to check: already fixed on master at 7d9d718 - the empty token matches the absence of a token, not a fallback to the gravity.

Test plan

  • 194 tests pass, including a new regression test for an outside edit
  • The trailing-whitespace case verified against live Mongo with the real repository code
  • The normal application path verified unchanged
  • Live data unchanged: 48 readings, 1 with advice
A fifth review of the PR, run against the pre-merge branch. One of its findings is a real hole worth closing even though the app cannot currently reach it. ## The finding `set_reading_advice` guards the write on the editing **token**. The token is written by `upsert_reading` on every edit, so every edit the application makes is caught. But a document edited **outside** the application - by a script, or by hand in a database tool - keeps its empty token. The reviewer reproduced it against live Mongo: ``` pre-token reading token: None -> stale thread carries "" fresh write matched: 1 stale write matched: 1 <<< stale advice overwrote the fresh one final advice: 'STALE ADVICE' ``` That edit bypassed `upsert_reading`, so no token was written. ## The fix The advice is generated from the comment **as well as** the gravity, so the comment is now a second, independent guard on the write. A stale write can only land if both the gravity and the comment are unchanged. ```python query["comment"] = { "$regex": "^" + re.escape(expected_comment.strip()) + r"\s*$" } ``` The match is a stripped, anchored regex rather than an exact one, because the stored comment and the one the view sees can differ by trailing whitespace - two of your live readings are padded (`'First time bottling '`). ## Verified against live Mongo With the real repository code: ``` stored comment has trailing whitespace: 'First time bottling ' write with the stripped comment matched: 1 <- the guard is not too strict stale write after an outside edit matched: 0 <- correctly refused FINAL advice: 'FRESH' | comment: 'edited outside' ``` And the normal application path is unaffected - a write with the real token and the current comment still lands. ## Also in this PR - The log message said the reading *"no longer holds the gravity it was written for"*, but the guard is the token and the comment. It now says what actually happened. ## The review's other findings - **LOW - the "all working" branch could empty a legitimate advice reply** and **LOW - single-paragraph pure working is still stored as advice**: both describe `_looks_like_advice` and `_ADVICE_MARKERS`, which **do not exist in `master`**. The marker-word heuristics were replaced by the deterministic `ADVICE:` marker split in `75f94d4`, and `master` has neither function. - **The empty-token hole it was asked to check**: already fixed on `master` at `7d9d718` - the empty token matches the *absence* of a token, not a fallback to the gravity. ## Test plan - [x] 194 tests pass, including a new regression test for an outside edit - [x] The trailing-whitespace case verified against live Mongo with the real repository code - [x] The normal application path verified unchanged - [x] Live data unchanged: 48 readings, 1 with advice
hermes added 1 commit 2026-11-09 22:26:54 +10:00
The editing token catches every edit the application makes, because
upsert_reading writes a fresh token on every write.  A document edited
outside the application - by a script, or by hand - keeps its empty
token, so a stale thread's write would still match on the gravity alone
and could put advice about the old comment back.

The advice is generated from the comment as well as the gravity, so the
comment is now a second, independent guard on the write.  A stale write
can only land if BOTH the gravity and the comment are unchanged.

The stored comment and the one the view sees can differ by trailing
whitespace - two of the live readings are padded - so the match is a
stripped, anchored regex rather than an exact one.

Also: the log message said the reading 'no longer holds the gravity it
was written for', but the guard is the token and the comment.  It now
says what actually happened.

194 tests pass.
Owner

PR received — starting review, sit tight 🫡

PR received — starting review, sit tight :saluting_face:
hermes merged commit cd45cc2f17 into master 2026-11-09 22:27:01 +10:00
Owner

PR Review Results

Combined Synthesis Review Report: fix(advice): guard the advice write on the comment as well as the token

1. Executive Summary

The objective of this PR is to resolve a data integrity vulnerability in the set_reading_advice function. The current implementation relies on an "editing token" for optimistic locking; however, edits made outside the application (e.g., via DB scripts) bypass this token, allowing stale application processes to overwrite fresh data.

The proposed solution implements a Multi-Factor Optimistic Locking strategy by expanding the write-guard to include both the token and the comment content. To handle existing data inconsistencies (trailing whitespace), the developer implemented an anchored, escaped regular expression.

All three review domains (Code, Security, and Infrastructure) have reached a consensus: The solution is technically sound, secures the identified hole, and follows security best practices regarding input sanitization.


2. Prioritized List of Issues

Severity Issue Domain Description
Low Data Normalization Gap Code/Infra The use of regex to handle trailing whitespace is a workaround for a root cause: inconsistent data storage.
Low Performance Overhead Infra Regex matching is computationally more expensive than direct equality checks, though negligible at current scale.
Low Observability Gap Security/Infra "Refused writes" due to outside edits are high-value signals that should be explicitly monitored/warned.

3. Domain-Specific Recommendations

🛠 Code Quality & Maintainability

  • Normalization at Ingress: Move away from "per-query" regex fixes. Create a backlog task to normalize (strip) comments before they are saved to the database.
  • Explicit Documentation: Add a code comment explicitly stating that the comment field acts as a fallback guard when the token is null to account for external database edits.

🛡 Security

  • Audit Logging: Elevate "stale write" rejection logs to a WARNING level and ensure they are routed to a centralized logging system. Frequent failures here could indicate either an operational error or an unauthorized external process attempting to manipulate data.
  • Token Verification: Ensure that if the "token" is ever exposed to a client-side interface, it is generated using a cryptographically secure method to prevent predictability.

☁️ Infrastructure & Performance

  • Indexing Strategy: Verify that the comment field is indexed. Since the regex is anchored (^), MongoDB can perform an efficient prefix scan, but only if an index is present.
  • Migration Script: Implement a one-time data migration script to .strip() all existing comments in the collection. This allows the application to revert to simple equality checks ({"comment": expected_comment}), improving query performance and readability.

4. Positive Aspects of the PR

  • Robust Sanitization: The use of re.escape() demonstrates senior-level attention to detail, successfully neutralizing potential Regular Expression Denial of Service (ReDoS) or regex injection attacks.
  • Correct Concurrency Pattern: Implementation of Optimistic Concurrency Control (OCC) is the industry-standard approach for preventing "lost updates" in NoSQL environments.
  • Improved Observability: The correction of log messages from confusing terminology ("gravity") to accurate failure reasons significantly reduces future MTTR (Mean Time to Recovery).
  • Rigorous Testing: The addition of a specific regression test for "outside edits" ensures the vulnerability does not reappear in future releases.

5. Overall Assessment & Recommendation

Final Rating: APPROVED

The PR successfully closes a critical logic hole. While there are opportunities for long-term architectural improvements regarding data normalization, the current implementation is a pragmatic, safe, and effective fix for the immediate problem. The security posture is improved, and the technical risks are minimal.

Recommendation: Merge as is, and create a follow-up ticket for data normalization and index verification.

## PR Review Results # Combined Synthesis Review Report: `fix(advice): guard the advice write on the comment as well as the token` ## 1. Executive Summary The objective of this PR is to resolve a data integrity vulnerability in the `set_reading_advice` function. The current implementation relies on an "editing token" for optimistic locking; however, edits made outside the application (e.g., via DB scripts) bypass this token, allowing stale application processes to overwrite fresh data. The proposed solution implements a **Multi-Factor Optimistic Locking** strategy by expanding the write-guard to include both the `token` and the `comment` content. To handle existing data inconsistencies (trailing whitespace), the developer implemented an anchored, escaped regular expression. All three review domains (Code, Security, and Infrastructure) have reached a consensus: **The solution is technically sound, secures the identified hole, and follows security best practices regarding input sanitization.** --- ## 2. Prioritized List of Issues | Severity | Issue | Domain | Description | | :--- | :--- | :--- | :--- | | **Low** | Data Normalization Gap | Code/Infra | The use of regex to handle trailing whitespace is a workaround for a root cause: inconsistent data storage. | | **Low** | Performance Overhead | Infra | Regex matching is computationally more expensive than direct equality checks, though negligible at current scale. | | **Low** | Observability Gap | Security/Infra | "Refused writes" due to outside edits are high-value signals that should be explicitly monitored/warned. | --- ## 3. Domain-Specific Recommendations ### 🛠 Code Quality & Maintainability * **Normalization at Ingress:** Move away from "per-query" regex fixes. Create a backlog task to normalize (strip) comments before they are saved to the database. * **Explicit Documentation:** Add a code comment explicitly stating that the `comment` field acts as a fallback guard when the `token` is null to account for external database edits. ### 🛡 Security * **Audit Logging:** Elevate "stale write" rejection logs to a `WARNING` level and ensure they are routed to a centralized logging system. Frequent failures here could indicate either an operational error or an unauthorized external process attempting to manipulate data. * **Token Verification:** Ensure that if the "token" is ever exposed to a client-side interface, it is generated using a cryptographically secure method to prevent predictability. ### ☁️ Infrastructure & Performance * **Indexing Strategy:** Verify that the `comment` field is indexed. Since the regex is anchored (`^`), MongoDB can perform an efficient prefix scan, but only if an index is present. * **Migration Script:** Implement a one-time data migration script to `.strip()` all existing comments in the collection. This allows the application to revert to simple equality checks (`{"comment": expected_comment}`), improving query performance and readability. --- ## 4. Positive Aspects of the PR * **Robust Sanitization:** The use of `re.escape()` demonstrates senior-level attention to detail, successfully neutralizing potential Regular Expression Denial of Service (ReDoS) or regex injection attacks. * **Correct Concurrency Pattern:** Implementation of Optimistic Concurrency Control (OCC) is the industry-standard approach for preventing "lost updates" in NoSQL environments. * **Improved Observability:** The correction of log messages from confusing terminology ("gravity") to accurate failure reasons significantly reduces future MTTR (Mean Time to Recovery). * **Rigorous Testing:** The addition of a specific regression test for "outside edits" ensures the vulnerability does not reappear in future releases. --- ## 5. Overall Assessment & Recommendation **Final Rating: APPROVED** The PR successfully closes a critical logic hole. While there are opportunities for long-term architectural improvements regarding data normalization, the current implementation is a pragmatic, safe, and effective fix for the immediate problem. The security posture is improved, and the technical risks are minimal. **Recommendation:** Merge as is, and create a follow-up ticket for data normalization and index verification.
Sign in to join this conversation.
No Reviewers
No labels
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: armistace/beer_data#21