atheos-blog-lxc/RUNBOOK.md

RUNBOOK — atheos-blog-lxc

Atheos browser editor for armistace/blog, running in an Alpine LXC on Proxmox.

Guest	LXC 136 on node scotty (192.168.178.69)
IP	http://192.168.178.57/
OS	Alpine 3.23.2 · unprivileged · 2 vCPU · 1 GB RAM · 20 GB root · onboot=1
App	Atheos IDE, pinned to commit f90cd1806dbe0e25673a77e27390e8b1a240c3f6 (build 611)
Repo	git.aridgwayweb.com/hermes/atheos-blog-lxc (this provisioning repo)
Workspace	/var/www/atheos/workspace/blog → clone of armistace/blog
Access	LAN only. No NPM proxy host, no DNS record, no port-forward.

---

⚠️ STEP 1 — DO THIS FIRST: set your git identity

The agent deliberately did not configure a git identity (design decision D1, requirement FR-11). Git will refuse to commit until you set one. This is your first action after SSH-ing in.

ssh root@192.168.178.57

# become the user that runs PHP and owns the workspace
su -s /bin/sh atheos

# set YOUR identity (these values are placeholders - use your real ones)
git config --global user.name  "Andrew Ridgway"
git config --global user.email "ar17787@gmail.com"

# verify
git config --global --get user.name
git config --global --get user.email

Why you and not the agent: you own the armistace identity. Configuring it under your name programmatically would impersonate you and blur commit attribution.

The same identity must be set for the deploy key's user, which is the same account (atheos), so the single su above covers both the editor's commits and anything you do by hand.

---

STEP 2 — install the deploy key (Phase 2 prerequisite)

The guest already has a per-repo deploy keypair. Only the public half is shared:

# print it again if you need it
cat /home/atheos/.ssh/id_ed25519_blog.pub

Add it in Gitea:

1. Open armistace/blog → Settings → Deploy Keys → Add Deploy Key
2. Title: atheos-blog-lxc
3. Paste the public key
4. ✅ TICK "Allow write access" — required, otherwise push is rejected

Public key (as generated 2026-09-29):

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINj03Tep9pULTvVDCC3q1ZtocKTZSmDhPsB61Zic1SPk atheos-blog-lxc deploy key (armistace/blog)

Fingerprint to confirm you pasted the right one:

SHA256:qHvsjjSX79aPZRoNTRI4j+HH3hArAuINIRu9ewxx7t8  (ED25519)

---

STEP 3 — switch the remote to SSH (Phase 2)

Once the key is installed, tell the agent (or run it yourself):

su -s /bin/sh atheos -c \
  'git -C /var/www/atheos/workspace/blog remote set-url origin gitea-blog:armistace/blog.git'

# verify the transport actually authenticates
su -s /bin/sh atheos -c 'ssh -T gitea-blog'      # expect a Gitea greeting, not "Permission denied"
su -s /bin/sh atheos -c 'git -C /var/www/atheos/workspace/blog fetch --dry-run'

Why the remote uses gitea-blog and not a hostname

gitea-blog is an SSH config alias defined in /home/atheos/.ssh/config:

Host gitea-blog
    HostName 192.168.178.155      # LAN address
    User gitea                    # <-- NOT `git`! see the trap below
    IdentityFile ~/.ssh/id_ed25519_blog
    IdentitiesOnly yes
    Port 22

⚠️ TRAP: the SSH user is gitea, not git

Every GitHub/Gitea tutorial uses git@host. This instance does not. Its app.ini sets SSH_USER = gitea.

If you use User git, sshd replies Invalid user git and rejects the connection before it ever reads authorized_keys — so the error looks exactly like a rejected key and sends you hunting for a problem that isn't there. Confirmed in the sshd log on LXC 109:

Invalid user git from 192.168.178.57 port 39780
Connection closed by invalid user git 192.168.178.57 [preauth]

Correct user → Hi there! You've successfully authenticated with the deploy key named atheos-blog-lxc.

Two reasons for the alias:

1. IdentitiesOnly yes means no other key is ever offered — a stray git push cannot authenticate with a different (possibly broader) identity.

2. It's the LAN address because that's the only path that works. Verified 2026-09-29:

   Endpoint	Result
   git.aridgwayweb.com:22	unreachable — the public path does not forward :22 (only :443 via NPM)
   192.168.178.155:22	OPEN — Gitea's app.ini sets SSH_DOMAIN = 192.168.178.155, SSH_PORT = 22, START_SSH_SERVER = false

   So Gitea serves SSH from the LXC's own sshd and advertises the LAN address. Since this guest is LAN-only, LAN transport is also the correct choice — no internet round-trip.

---

STEP 4 — complete the Atheos installer

Open http://192.168.178.57/ in a browser. The Atheos first-run installer appears and creates the admin user — this is yours to own; the agent seeded no accounts.

Verify afterwards:

• you can open the Workspace and see a blog project
• opening src/content/*.md shows your posts
• saving persists (the files are at /var/www/atheos/workspace/blog)

Atheos git settings

Open Atheos → (user menu) → Settings to confirm the git integration. The codegit component handles commit/push. If you see "detected dubious ownership", the safe.directory entries are missing — restore them with:

su -s /bin/sh atheos -c 'git config --global --add safe.directory /var/www/atheos/workspace'
su -s /bin/sh atheos -c 'git config --global --add safe.directory /var/www/atheos/workspace/blog'

---

Day-to-day operations

Edit and commit

Do everything in the browser at http://192.168.178.57/. Commit via Atheos's git panel, or by hand:

ssh root@192.168.178.57
su -s /bin/sh atheos
cd /var/www/atheos/workspace/blog
git status
git add -A && git commit -m "post: <what changed>"
git push

The build is NOT here

This box only writes markdown. Building/publishing the Pelican site happens in the existing pipeline elsewhere. There is deliberately no Python, no pelican, no make on this guest.

Services (OpenRC — not systemd)

rc-service nginx status
rc-service php-fpm83 status
rc-update show | grep -E 'nginx|php-fpm83'     # both must be in 'default'

Logs

tail -f /var/log/nginx/atheos.error.log        # PHP fatals surface here
tail -f /var/log/nginx/atheos.access.log

Reboot / recovery

pct reboot 136      # from scotty, or
pct stop 136 && pct start 136

Services are enabled at the OpenRC default runlevel, so they return automatically (verified — see aidlc-docs acceptance criteria AC-6).

Rebuild from scratch

The whole guest is reproducible:

# on scotty (192.168.178.69), as root
pct destroy 136
sh scripts/provision.sh        # from this repo, on the node

Nothing unique lives on the guest: the blog's source of truth is Gitea. Only uncommitted edits would be lost — commit often.

Update Atheos (and re-pin)

Atheos has no releases, so this install is pinned to a commit. To move to the latest upstream commit, install it, and re-pin:

sh scripts/update-atheos.sh       # on the guest; preserves workspace/, backs up, verifies, re-pins

---

Security model (summary — full detail in docs/security-notes.md)

• Git auth = a per-repository deploy key, scoped to armistace/blog, write-enabled. Not a PAT (whose write:repository scope reaches every repo the account can write).
• Private key: /home/atheos/.ssh/id_ed25519_blog, mode 600, owned by the php-fpm user.
• The web app and the key share one unprivileged user, so a compromised Atheos can write blog. That is the accepted floor — one repo, and (with branch protection) one branch.
• LAN-only. If this is ever exposed: keep branch protection on blog's master, terminate TLS at NPM, and put authentication in front of Atheos. See docs/security-notes.md §2 (S-1…S-14).

Container resource reporting (a real trap — cost debugging time)

uptime and free on Alpine are busybox applets. Busybox reads via the sysinfo() syscall, which lxcfs cannot intercept — so inside this container they report the PVE host, not the container. A container that restarted seconds ago will cheerfully claim "up 2 days".

lxcfs is working correctly here — the virtualised files are right:

/proc/meminfo → MemTotal: 1048576 kB   ← our 1024 MB limit
/proc/uptime  → 105 s                  ← container
/proc/cpuinfo → 2 processors           ← container

The container is correct; the tool is wrong. Fix = apk add procps-ng, whose /usr/bin/free and /usr/bin/uptime read those virtualised /proc files:

Tool	Reports
procps-ng free	1024 MB	✅ correct
busybox free	15701 MB	❌ host
procps-ng uptime	up 2 min	✅ correct
busybox uptime	up 2 days	❌ host

Note: apk add lxcfs inside the guest is not a fix — lxcfs is a host-provided FUSE filesystem, not a guest package. provision.sh installs procps-ng for this reason. (df is unaffected: it stats the filesystem directly, so it was always accurate.)

---

Reaching the guest when SSH from your laptop fails

The guest has root SSH enabled with your fleet keys installed:

• ssh-rsa ... aridgway@strategenics-25
• ssh-ed25519 ... oct2025

If those keys aren't what you're using, add/push a key via the PVE node (which always works):

ssh root@192.168.178.69 "pct exec 136 -- sh -c 'echo \"<your-pubkey>\" >> /root/.ssh/authorized_keys'"

---

Troubleshooting

Symptom	Cause	Fix
502 Bad Gateway	nginx fastcgi_pass and the FPM pool listen disagree	grep -E '^listen' /etc/php83/php-fpm.d/www.conf and compare with the fastcgi_pass line in /etc/nginx/http.d/atheos.conf. Both must read /run/php-fpm83/php-fpm.sock. Then restart both services.
"detected dubious ownership"	safe.directory missing or the repo is owned by a different user	Re-add the two safe.directory entries (STEP 4)
git commit refuses: "please tell me who you are"	identity not set — expected on a fresh build	STEP 1
git push: Permission denied (publickey)	deploy key not installed, or the SSH user is wrong, or "Allow write access" not ticked	Check all three. Especially: grep User /home/atheos/.ssh/config must say gitea, not git (see STEP 3 trap). On the Gitea host, confirm with journalctl -u ssh | tail — Invalid user git means the username is wrong, not the key.
push times out	remote still pointing at git.aridgwayweb.com (:22 not forwarded)	STEP 3 — use the gitea-blog alias
Atheos shows a blank page / 500	PHP fatal	tail -50 /var/log/nginx/atheos.error.log
Atheos installer says a path isn't writable	ownership broken	chown -R atheos:atheos /var/www/atheos
free / uptime show the PVE host's memory or days of uptime	on Alpine these are busybox applets using the sysinfo() syscall, which lxcfs cannot intercept	apk add procps-ng — its /usr/bin/free and /usr/bin/uptime read the lxcfs-virtualised /proc files. Already installed by provision.sh; this bites if you rebuild by hand.
PHP session errors	rootfs full	df -h / (the rootfs is 20 GB and the app is tiny, so this indicates something else)

---

Change log

Date	Change
2026-09-29	Initial build: LXC 136 on scotty, Alpine 3.23.2, nginx + php83-fpm (socket), Atheos pinned at f90cd18 (build 611), armistace/blog cloned over HTTPS, per-repo deploy key generated. LAN-only.
2026-09-29	Added the writer-branch workflow: 6 macros installed into Atheos (start/join, commit+push, destructive reset-to-master, status, refresh master, merge master) writing to a long-lived atheos-writer branch; merging stays in Gitea. See docs/writer-branch-workflow.md.

Writer-branch workflow (quick reference)

The editor writes to atheos-writer; you merge to master in Gitea.

Macros (right-click the blog folder → Macros):

#	Macro	Purpose
1	Start/Join Writer Branch	creates or re-joins atheos-writer (idempotent)
2	Commit + Push Writer Branch	the everyday button
3	DESTRUCTIVE: Reset Writer Branch to Master	wipes writer's commits, keeps your files
4	Status + Recent Log	read-only
5	Refresh Master (ff-only)	pull master's merges down
6	Merge Master Into Writer Branch	when master has moved ahead

⚠️ Requires branch protection on armistace/blog's master — it is not currently enabled, so the deploy key can force-push master directly. See docs/writer-branch-workflow.md §Setup. ⚠️ Macro 2 uses git add -A — everything left in the repo gets published to Gitea. Keep scratch files outside workspace/blog/.

Full detail, recovery steps and caveats: docs/writer-branch-workflow.md.