Files
atheos-blog-lxc/docs/resiliency-notes.md
T
hermes c2897ed660 feat: provision Alpine LXC 136 running Atheos, wired to Gitea
Provisions a LAN-only browser code editor (Atheos) in an Alpine 3.23 LXC on the
scotty PVE node, with armistace/blog cloned and ready to edit. Built via the
AI-DLC workflow; the full process trail is in aidlc-docs/.

Deliverables
- scripts/provision.sh          rebuild the guest from scratch (re-runnable)
- scripts/update-atheos.sh      move Atheos to latest upstream and RE-PIN
- scripts/build-and-test.sh     captures evidence for every acceptance criterion
- files/atheos.conf             nginx vhost (socket path byte-matches the FPM pool)
- files/ssh-config              ~atheos/.ssh/config deploy-key alias
- RUNBOOK.md                    operational guide (START AT STEP 1)
- docs/security-notes.md        credential model + wider-web hardening TODOs
- docs/resiliency-notes.md      what a non-PoC deployment would need

Design decisions
- D1 no git identity is configured by the agent; the operator sets it. Git will
  refuse the first commit until then, which is deliberate (requirement FR-11).
- D2 Atheos pinned to commit f90cd18 (build 611) with a verified tarball sha256,
  plus update-atheos.sh as the deliberate escape hatch.
- D3 nginx :80, web root /var/www/atheos, php-fpm over a unix socket.
- D6 git auth is a per-repository DEPLOY KEY scoped to armistace/blog, not a
  personal access token (a PAT's write:repository scope reaches every repo the
  account can write).

Verified on the live guest (40 checks, 0 failures)
- AC-2  Atheos served over HTTP 200 by nginx+php-fpm; socket paths byte-match
- AC-3  20 posts + themes/cleanblog present; owned by the php-fpm user
- AC-4  all five writable paths writable as the service user
- AC-5  remote targets armistace/blog; deploy key authenticates over SSH
- AC-6  survival proven with a real container restart, marker byte-identical
- AC-8  no python/pelican/pip/make and no DB engine - scope boundary enforced
- FR-11 git identity is empty; deploy key mode 600 owned by the service user

Traps documented in the RUNBOOK (each cost real debugging time)
- The Gitea SSH user is `gitea`, NOT `git` (app.ini SSH_USER=gitea). With `git`,
  sshd answers "Invalid user git" and rejects before reading authorized_keys,
  which is indistinguishable from a bad key.
- The Gitea SSH endpoint is the LAN address (:22 is not forwarded publicly).
- `uptime` and `free` report the PVE HOST inside an LXC without lxcfs, so a
  container that just restarted can claim days of uptime.
2026-09-29 15:44:11 +10:00

3.1 KiB

Resiliency Notes — atheos-blog-lxc

Status: resiliency-baseline extension opted out (LAN-only PoC) per Andrew's standing preference. Purpose: record what a non-PoC deployment would need, so the trail survives.


1. What this guest is, from a resiliency standpoint

A single small container that can be destroyed and rebuilt — that is the design, not a gap. armistace/blog is the durability anchor: the source of truth lives in Gitea, not on this guest. Losing the LXC loses only:

  • the Atheos install (reproducible: scripts/provision.sh)
  • the local clone (re-clonable)
  • uncommitted edits ← the only genuinely perishable data

2. Current posture

Aspect State Note
Rootfs 20 GB, thin-provisioned on scotty-local-lvm thin pool had 771 GB free at design time
onboot enabled survives a node restart
Backups none acceptable: nothing unique is stored
Snapshots none taken (a pre-* snapshot is used during upgrades only)
Rebuild path scripts/provision.sh this is the resiliency mechanism
Reboot survival verified, not assumed AC-6 — service returns and the clone persists

3. If this ever moves beyond a LAN PoC

# Item Touch-point
R-1 Scheduled snapshot of the guest (pct snapshot / PBS) with retention PVE / PBS
R-2 Automated verification of the rebuild path — run provision.sh into a throwaway VMID on a schedule; a backup you have never restored is a claim CI or cron on the PVE node
R-3 Alert on guest down (pct status + HTTP probe) observability stack
R-4 Push-based durability for in-progress work: an auto-commit/ahead-of-time branch so uncommitted edits stop being the failure mode a small hook or Atheos-side job
R-5 Disk alert at 80% of the 20 GB rootfs (a full rootfs silently breaks PHP sessions) observability stack
R-6 Move the clone's durability to the repo: rely on Gitea backup/restore rather than guest backups Gitea
R-7 Document RTO/RPO: here RTO ≈ re-run provision.sh (minutes); RPO = last commit RUNBOOK
R-8 scotty-local-lvm thin-pool monitoring — a thin pool that fills takes down every guest on it, not just this one PVE node

4. Design choices that already help

  • Nothing unique is stored on the guest — the deliberately tiny 20 GB root is a feature: it bounds what can be lost and keeps rebuild cheap.
  • Provisioning is a script, so rebuild is not a memory exercise (FR-9).
  • The build pipeline is elsewhere (Q1=A), so this guest is not on the critical path for publishing.
  • OpenRC services are enabled at boot (AC-6 verifies it), so a reboot is not a repair event.

5. Accepted risks

Risk Why accepted Revisit trigger
No backups no unique data; rebuild is scripted if the guest starts holding anything unique
Single guest on a single node LAN PoC scope on any production expectation → R-1/R-3
Uncommitted edits are perishable normal for a git worktree; operator controls commit cadence if it bites → R-4