gitea-repo-template/RUNBOOK.md

# Gitea Repo Template — Operations Runbook

This is the **cookie-cutter** for new homelab repos. On Gitea you create a new
repo **from this template** and it copies `.gitea/`, `kube/`, `ci/buildkit/`,
`scripts/`, and this RUNBOOK. Fill in the `<PLACEHOLDERS>` per service, then
follow the bootstrap flow below. Everything documented here is proven live on
`armistace/wedding-photos`.

---

## 1. Create a repo from this template

- Gitea UI: `+` → *New Repository* → **"Copy to new repository"** source = `hermes/gitea-repo-template`.
- Bot/automation: `POST /api/v1/repos/{owner}/{repo}/generate` (See `~/.hermes/scripts/gitea-bootstrap-new-repo.sh`).

After creation, commit+push a branch filling in:
- `.gitea/workflows/build_push.yml` → `<APP>`, `<OWNER>`, `<NS>`, `<KEY_*>`
- `scripts/reconcile-cluster-inject.sh` → same placeholders
- `kube/*` , `ci/buildkit/*` → your image/service names
- This RUNBOOK's placeholders → actual namespace/app

---

## 2. One-time per-repo setup (do this once, after the first commit)

### 2.1 Repo secrets + vars (Gitea Settings → Actions/secrets)
Set on the repo (or reuse the global `armistace`-scoped ones):
- `KUBEC_CONFIG_BUILDX_NEW_2` — kubeconfig for the cluster (buildx secret)
- `REG_PASSWORD` — Gitea registry password for `<OWNER>`
- every `<KEY_*>` referenced by `reconcile-cluster-inject.sh` SECRET_KEYS
- vars for CFG: e.g. `DB_HOST`, `S3_ENDPOINT_URL`, etc.

### 2.2 Branch protection (hard guard, server-side)
Created automatically by the bootstrap script (agent runs it with the bot token
on `hermes/`-owned repos). Rule on `master`:
- `enable_push + enable_push_whitelist=true`, push whitelist `[armistace]`
  → only Andrew can DIRECT-push master; agent direct push is blocked even with `--no-verify`
- `enable_merge_whitelist=true`, merge whitelist `[hermes, armistace]`
  → agent can still MERGE PRs (merge ≠ direct push)

### 2.3 Local pre-commit guard
Point the repo at the shared git hook so it blocks direct-to-master commits:
```bash
git config core.hooksPath ~/dev/git-hooks
```
(Shared hook lives at `~/dev/git-hooks/pre-commit` on the control host; the
master-guard binary is bundled in the `gitea-pr-workflow` skill.)

---

## 3. Architecture & storage

### 3.1 Cluster injection via Gitea repo secrets + vars
`scripts/reconcile-cluster-inject.sh` reconciles the live Secret/ConfigMap each
deploy with three rules (priority order):
1. Gitea supplies a **non-empty** value → that value wins (seed / rotate).
2. Key **missing** from the live resource → write placeholder (env ref always resolves).
3. Otherwise → **preserve** the live value (hand-provisioned creds never clobbered).

The patch is built by Python `json.dumps` (no shell interpolation). Add every
key the deployments reference to `SECRET_KEYS` / `CM_KEYS`; list new repo
secrets/vars here so a handoff never loses them.

### 3.2 Persistent buildkit cache (optional, durable CI cache)
`ci/buildkit/` installs a single-replica Longhorn-backed buildkit daemon so CI
image builds reuse cache. Manifests are shipped but NOT auto-applied — apply
once:
```bash
kubectl apply -f ci/buildkit/01-storageclass.yaml
kubectl apply -f ci/buildkit/02-statefulset.yaml
kubectl apply -f ci/buildkit/03-service.yaml
```
The remote-driver workflow points at `buildkit.gitea-runner.svc:1234`.
Disk watchdog + revert scripts live on the control host
(`~/.hermes/scripts/buildkit-cache-monitor.sh`, `buildkit-re-enable.sh`) — NOT
in this repo (ops tooling doesn't go in app repos).

---

## 4. Deploying / re-deploying from scratch

### 4.1 Provision the namespace + live Secret ONCE by hand
CI will `kubectl apply` manifests and reconcile the Secret, but the FIRST seed
of real credentials is manual (CI only writes placeholders when a key is absent):
```bash
kubectl create namespace <NS>
kubectl create secret docker-registry regcred \
  --docker-server=git.aridgwayweb.com --docker-username=<OWNER> \
  --docker-password='<REG_PASSWORD>' --docker-email=<OWNER>@aridgwayweb.com \
  -n <NS> --dry-run=client -o yaml | kubectl apply -f -
kubectl -n <NS> create secret generic <NS>-secret \
  --from-literal=<KEY_1>='<real-value>' \
  --from-literal=<KEY_2>='<real-value>'
```

### 4.2 Set the ConfigMap
```bash
kubectl -n <NS> apply -f kube/<NS>_configmap.yaml
```

### 4.3 After CI deploy — verify
```bash
kubectl -n <NS> get pods
kubectl -n <NS> rollout status deployment/<APP>-backend --timeout=180s
```

---

## 5. Common operations

- **Rotate a secret**: update the Gitea repo secret → push an empty commit or
  re-run `workflow_dispatch` → reconcile applies the new value → `kubectl -n <NS> rollout restart deployment/<APP>-backend`.
- **If pods CrashLoopBackOff after a secret change**: the env ref is likely
  unresolvable — check `kubectl get event` / secret keys; add the missing key to
  `reconcile-cluster-inject.sh` SECRET_KEYS and the workflow export block.
- **Never delete the namespace** on deploy (CI does not; don't start) — it
  destroys the live Secret.

---

## 6. Troubleshooting

- **Build-deploy fails ~36s, "no remote endpoint provided"**: the buildx
  `endpoint` was put under `driver-opts`. Move it to a top-level `with:` input.
- **Multi-arch build OOMs/crashes the buildkit pod**: build `linux/amd64` only.
- **`COPY --from=ghcr.io/...` TLS timeout**: buildkit can't reach ghcr.io. Use
  an ECR base + `pip install uv`.
- **PR files API returns empty `patch` fields**: build review payload from local
  `git diff` — never trust the API `patch`.

---

## 7. Handoff checklist

- [ ] All `<PLACEHOLDERS>` filled (workflow, reconcile script, kube/, this RUNBOOK)
- [ ] Repo secrets + vars set; keys match `reconcile-cluster-inject.sh` + workflow `export` block
- [ ] Branch protection on `master` enabled (push whitelist `[armistace]`, merge whitelist `[hermes, armistace]`)
- [ ] `git config core.hooksPath ~/dev/git-hooks` set in the new repo
- [ ] First deploy: namespace seeded with a real Secret (RUNBOOK §4.1) before relying on CI reconcile
- [ ] Any new env key added to §3.1 table + §2.1 list so a handoff doesn't lose it