atheos-blog-lxc/RUNBOOK.md

# RUNBOOK — atheos-blog-lxc

Atheos browser editor for `armistace/blog`, running in an Alpine LXC on Proxmox.

| | |
|---|---|
| **Guest** | LXC **136** on node `scotty` (192.168.178.69) |
| **IP** | **http://192.168.178.57/** |
| **OS** | Alpine 3.23.2 · unprivileged · 2 vCPU · 1 GB RAM · 20 GB root · `onboot=1` |
| **App** | Atheos IDE, pinned to commit `f90cd1806dbe0e25673a77e27390e8b1a240c3f6` (build 611) |
| **Repo** | `git.aridgwayweb.com/hermes/atheos-blog-lxc` (this provisioning repo) |
| **Workspace** | `/var/www/atheos/workspace/blog` → clone of `armistace/blog` |
| **Access** | LAN only. No NPM proxy host, no DNS record, no port-forward. |

---

## ⚠️ STEP 1 — DO THIS FIRST: set your git identity

**The agent deliberately did not configure a git identity** (design decision D1, requirement FR-11).
Git will **refuse to commit** until you set one. This is your first action after SSH-ing in.

```sh
ssh root@192.168.178.57

# become the user that runs PHP and owns the workspace
su -s /bin/sh atheos

# set YOUR identity (these values are placeholders - use your real ones)
git config --global user.name  "Andrew Ridgway"
git config --global user.email "ar17787@gmail.com"

# verify
git config --global --get user.name
git config --global --get user.email
```

> Why you and not the agent: you own the `armistace` identity. Configuring it under your name
> programmatically would impersonate you and blur commit attribution.

The same identity must be set for the deploy key's user, which is the **same account** (`atheos`),
so the single `su` above covers both the editor's commits and anything you do by hand.

---

## STEP 2 — install the deploy key (Phase 2 prerequisite)

The guest already has a per-repo deploy keypair. Only the **public** half is shared:

```sh
# print it again if you need it
cat /home/atheos/.ssh/id_ed25519_blog.pub
```

Add it in Gitea:

1. Open `armistace/blog` → **Settings** → **Deploy Keys** → **Add Deploy Key**
2. Title: `atheos-blog-lxc`
3. Paste the public key
4. ✅ **TICK "Allow write access"** — required, otherwise push is rejected

Public key (as generated 2026-09-29):

```
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINj03Tep9pULTvVDCC3q1ZtocKTZSmDhPsB61Zic1SPk atheos-blog-lxc deploy key (armistace/blog)
```

Fingerprint to confirm you pasted the right one:

```
SHA256:qHvsjjSX79aPZRoNTRI4j+HH3hArAuINIRu9ewxx7t8  (ED25519)
```

---

## STEP 3 — switch the remote to SSH (Phase 2)

Once the key is installed, tell the agent (or run it yourself):

```sh
su -s /bin/sh atheos -c \
  'git -C /var/www/atheos/workspace/blog remote set-url origin gitea-blog:armistace/blog.git'

# verify the transport actually authenticates
su -s /bin/sh atheos -c 'ssh -T gitea-blog'      # expect a Gitea greeting, not "Permission denied"
su -s /bin/sh atheos -c 'git -C /var/www/atheos/workspace/blog fetch --dry-run'
```

### Why the remote uses `gitea-blog` and not a hostname

`gitea-blog` is an SSH config alias defined in `/home/atheos/.ssh/config`:

```
Host gitea-blog
    HostName 192.168.178.155      # LAN address
    User gitea                    # <-- NOT `git`! see the trap below
    IdentityFile ~/.ssh/id_ed25519_blog
    IdentitiesOnly yes
    Port 22
```

> ### ⚠️ TRAP: the SSH user is `gitea`, not `git`
> Every GitHub/Gitea tutorial uses `git@host`. **This instance does not.** Its `app.ini` sets
> `SSH_USER = gitea`.
>
> If you use `User git`, sshd replies **`Invalid user git`** and rejects the connection
> **before it ever reads `authorized_keys`** — so the error looks exactly like a rejected key
> and sends you hunting for a problem that isn't there. Confirmed in the sshd log on LXC 109:
> ```
> Invalid user git from 192.168.178.57 port 39780
> Connection closed by invalid user git 192.168.178.57 [preauth]
> ```
> Correct user → `Hi there! You've successfully authenticated with the deploy key named atheos-blog-lxc`.

Two reasons for the alias:

1. **`IdentitiesOnly yes`** means no other key is ever offered — a stray `git push` cannot
   authenticate with a different (possibly broader) identity.
2. **It's the LAN address because that's the only path that works.** Verified 2026-09-29:

   | Endpoint | Result |
   |---|---|
   | `git.aridgwayweb.com:22` | **unreachable** — the public path does not forward :22 (only :443 via NPM) |
   | `192.168.178.155:22` | **OPEN** — Gitea's `app.ini` sets `SSH_DOMAIN = 192.168.178.155`, `SSH_PORT = 22`, `START_SSH_SERVER = false` |

   So Gitea serves SSH from the LXC's own `sshd` and advertises the LAN address. Since this guest is
   LAN-only, LAN transport is also the correct choice — no internet round-trip.

---

## STEP 4 — complete the Atheos installer

Open **http://192.168.178.57/** in a browser. The Atheos first-run installer appears and creates
the **admin user** — this is yours to own; the agent seeded no accounts.

Verify afterwards:

- you can open the **Workspace** and see a `blog` project
- opening `src/content/*.md` shows your posts
- saving persists (the files are at `/var/www/atheos/workspace/blog`)

### Atheos git settings

Open Atheos → (user menu) → **Settings** to confirm the git integration. The `codegit` component
handles commit/push. If you see *"detected dubious ownership"*, the `safe.directory` entries are
missing — restore them with:

```sh
su -s /bin/sh atheos -c 'git config --global --add safe.directory /var/www/atheos/workspace'
su -s /bin/sh atheos -c 'git config --global --add safe.directory /var/www/atheos/workspace/blog'
```

---

## Day-to-day operations

### Edit and commit

Do everything in the browser at `http://192.168.178.57/`. Commit via Atheos's git panel, or by hand:

```sh
ssh root@192.168.178.57
su -s /bin/sh atheos
cd /var/www/atheos/workspace/blog
git status
git add -A && git commit -m "post: <what changed>"
git push
```

### The build is NOT here

This box **only writes markdown**. Building/publishing the Pelican site happens in the existing
pipeline elsewhere. There is deliberately **no Python, no pelican, no `make`** on this guest.

### Services (OpenRC — not systemd)

```sh
rc-service nginx status
rc-service php-fpm83 status
rc-update show | grep -E 'nginx|php-fpm83'     # both must be in 'default'
```

### Logs

```sh
tail -f /var/log/nginx/atheos.error.log        # PHP fatals surface here
tail -f /var/log/nginx/atheos.access.log
```

### Reboot / recovery

```sh
pct reboot 136      # from scotty, or
pct stop 136 && pct start 136
```

Services are enabled at the OpenRC `default` runlevel, so they return automatically
(verified — see `aidlc-docs` acceptance criteria AC-6).

### Rebuild from scratch

The whole guest is reproducible:

```sh
# on scotty (192.168.178.69), as root
pct destroy 136
sh scripts/provision.sh        # from this repo, on the node
```

Nothing unique lives on the guest: the blog's source of truth is Gitea. Only **uncommitted edits**
would be lost — commit often.

### Update Atheos (and re-pin)

Atheos has no releases, so this install is pinned to a commit. To move to the latest upstream commit,
install it, and re-pin:

```sh
sh scripts/update-atheos.sh       # on the guest; preserves workspace/, backs up, verifies, re-pins
```

---

## Security model (summary — full detail in `docs/security-notes.md`)

- **Git auth = a per-repository deploy key**, scoped to `armistace/blog`, write-enabled.
  Not a PAT (whose `write:repository` scope reaches **every** repo the account can write).
- Private key: `/home/atheos/.ssh/id_ed25519_blog`, mode `600`, owned by the php-fpm user.
- The web app and the key share one unprivileged user, so a compromised Atheos **can** write `blog`.
  That is the accepted floor — one repo, and (with branch protection) one branch.
- **LAN-only.** If this is ever exposed: keep branch protection on `blog`'s `master`, terminate TLS
  at NPM, and put authentication *in front* of Atheos. See `docs/security-notes.md` §2 (S-1…S-14).

### Container resource reporting (a real trap — cost debugging time)

`uptime` and `free` on Alpine are **busybox** applets. Busybox reads via the **`sysinfo()` syscall**,
which lxcfs **cannot intercept** — so inside this container they report the **PVE host**, not the
container. A container that restarted seconds ago will cheerfully claim *"up 2 days"*.

lxcfs **is** working correctly here — the virtualised files are right:

```
/proc/meminfo → MemTotal: 1048576 kB   ← our 1024 MB limit
/proc/uptime  → 105 s                  ← container
/proc/cpuinfo → 2 processors           ← container
```

The container is correct; **the tool is wrong**. Fix = `apk add procps-ng`, whose `/usr/bin/free`
and `/usr/bin/uptime` read those virtualised `/proc` files:

| Tool | Reports | |
|---|---|---|
| `procps-ng` `free` | `1024 MB` | ✅ correct |
| busybox `free` | `15701 MB` | ❌ host |
| `procps-ng` `uptime` | `up 2 min` | ✅ correct |
| busybox `uptime` | `up 2 days` | ❌ host |

> **Note:** `apk add lxcfs` inside the guest is **not** a fix — lxcfs is a host-provided FUSE
> filesystem, not a guest package. `provision.sh` installs `procps-ng` for this reason.
> (`df` is unaffected: it stats the filesystem directly, so it was always accurate.)

---

## Reaching the guest when SSH from your laptop fails

The guest has **root SSH enabled** with your fleet keys installed:

- `ssh-rsa ... aridgway@strategenics-25`
- `ssh-ed25519 ... oct2025`

If those keys aren't what you're using, add/push a key via the PVE node (which always works):

```sh
ssh root@192.168.178.69 "pct exec 136 -- sh -c 'echo \"<your-pubkey>\" >> /root/.ssh/authorized_keys'"
```

---

## Troubleshooting

| Symptom | Cause | Fix |
|---|---|---|
| **502 Bad Gateway** | nginx `fastcgi_pass` and the FPM pool `listen` disagree | `grep -E '^listen' /etc/php83/php-fpm.d/www.conf` and compare with the `fastcgi_pass` line in `/etc/nginx/http.d/atheos.conf`. Both must read `/run/php-fpm83/php-fpm.sock`. Then restart both services. |
| **"detected dubious ownership"** | `safe.directory` missing or the repo is owned by a different user | Re-add the two `safe.directory` entries (STEP 4) |
| **git commit refuses: "please tell me who you are"** | identity not set — **expected on a fresh build** | STEP 1 |
| **git push: Permission denied (publickey)** | deploy key not installed, **or the SSH user is wrong**, or "Allow write access" not ticked | Check all three. Especially: `grep User /home/atheos/.ssh/config` must say **`gitea`**, not `git` (see STEP 3 trap). On the Gitea host, confirm with `journalctl -u ssh \| tail` — `Invalid user git` means the username is wrong, not the key. |
| **push times out** | remote still pointing at `git.aridgwayweb.com` (:22 not forwarded) | STEP 3 — use the `gitea-blog` alias |
| **Atheos shows a blank page / 500** | PHP fatal | `tail -50 /var/log/nginx/atheos.error.log` |
| **Atheos installer says a path isn't writable** | ownership broken | `chown -R atheos:atheos /var/www/atheos` |
| **`free` / `uptime` show the PVE host's memory or days of uptime** | on Alpine these are **busybox** applets using the `sysinfo()` **syscall**, which lxcfs cannot intercept | `apk add procps-ng` — its `/usr/bin/free` and `/usr/bin/uptime` read the lxcfs-virtualised `/proc` files. Already installed by `provision.sh`; this bites if you rebuild by hand. |
| **PHP session errors** | rootfs full | `df -h /` (the rootfs is 20 GB and the app is tiny, so this indicates something else) |

---

## Change log

| Date | Change |
|---|---|
| 2026-09-29 | Initial build: LXC 136 on scotty, Alpine 3.23.2, nginx + php83-fpm (socket), Atheos pinned at `f90cd18` (build 611), `armistace/blog` cloned over HTTPS, per-repo deploy key generated. LAN-only. |