gitea-repo-template/RUNBOOK.md

Gitea Repo Template — Operations Runbook

This is the cookie-cutter for new homelab repos. On Gitea you create a new repo from this template and it copies .gitea/, kube/, ci/buildkit/, scripts/, and this RUNBOOK. Fill in the <PLACEHOLDERS> per service, then follow the bootstrap flow below. Everything documented here is proven live on armistace/wedding-photos.

---

1. Create a repo from this template

• Gitea UI: + → New Repository → "Copy to new repository" source = hermes/gitea-repo-template.
• Bot/automation: POST /api/v1/repos/{owner}/{repo}/generate (See ~/.hermes/scripts/gitea-bootstrap-new-repo.sh).

After creation, commit+push a branch filling in:

• .gitea/workflows/build_push.yml → <APP>, <OWNER>, <NS>, <KEY_*>
• scripts/reconcile-cluster-inject.sh → same placeholders
• kube/* , ci/buildkit/* → your image/service names
• This RUNBOOK's placeholders → actual namespace/app

---

2. One-time per-repo setup (do this once, after the first commit)

2.1 Repo secrets + vars (Gitea Settings → Actions/secrets)

Set on the repo (or reuse the global armistace-scoped ones):

• KUBEC_CONFIG_BUILDX_NEW_2 — kubeconfig for the cluster (buildx secret)
• REG_PASSWORD — Gitea registry password for <OWNER>
• every <KEY_*> referenced by reconcile-cluster-inject.sh SECRET_KEYS
• vars for CFG: e.g. DB_HOST, S3_ENDPOINT_URL, etc.

2.2 Branch protection (hard guard, server-side)

Created automatically by the bootstrap script (agent runs it with the bot token on hermes/-owned repos). Rule on master:

• enable_push + enable_push_whitelist=true, push whitelist [armistace] → only Andrew can DIRECT-push master; agent direct push is blocked even with --no-verify
• enable_merge_whitelist=true, merge whitelist [hermes, armistace] → agent can still MERGE PRs (merge ≠ direct push)

2.3 Local pre-commit guard

Point the repo at the shared git hook so it blocks direct-to-master commits:

git config core.hooksPath ~/dev/git-hooks

(Shared hook lives at ~/dev/git-hooks/pre-commit on the control host; the master-guard binary is bundled in the gitea-pr-workflow skill.)

---

3. Architecture & storage

3.1 Cluster injection via Gitea repo secrets + vars

scripts/reconcile-cluster-inject.sh reconciles the live Secret/ConfigMap each deploy with three rules (priority order):

1. Gitea supplies a non-empty value → that value wins (seed / rotate).
2. Key missing from the live resource → write placeholder (env ref always resolves).
3. Otherwise → preserve the live value (hand-provisioned creds never clobbered).

The patch is built by Python json.dumps (no shell interpolation). Add every key the deployments reference to SECRET_KEYS / CM_KEYS; list new repo secrets/vars here so a handoff never loses them.

3.2 Persistent buildkit cache (optional, durable CI cache)

ci/buildkit/ installs a single-replica Longhorn-backed buildkit daemon so CI image builds reuse cache. Manifests are shipped but NOT auto-applied — apply once:

kubectl apply -f ci/buildkit/01-storageclass.yaml
kubectl apply -f ci/buildkit/02-statefulset.yaml
kubectl apply -f ci/buildkit/03-service.yaml

The remote-driver workflow points at buildkit.gitea-runner.svc:1234. Disk watchdog + revert scripts live on the control host (~/.hermes/scripts/buildkit-cache-monitor.sh, buildkit-re-enable.sh) — NOT in this repo (ops tooling doesn't go in app repos).

---

4. Deploying / re-deploying from scratch

4.1 Provision the namespace + live Secret ONCE by hand

CI will kubectl apply manifests and reconcile the Secret, but the FIRST seed of real credentials is manual (CI only writes placeholders when a key is absent):

kubectl create namespace <NS>
kubectl create secret docker-registry regcred \
  --docker-server=git.aridgwayweb.com --docker-username=<OWNER> \
  --docker-password='<REG_PASSWORD>' --docker-email=<OWNER>@aridgwayweb.com \
  -n <NS> --dry-run=client -o yaml | kubectl apply -f -
kubectl -n <NS> create secret generic <NS>-secret \
  --from-literal=<KEY_1>='<real-value>' \
  --from-literal=<KEY_2>='<real-value>'

4.2 Set the ConfigMap

kubectl -n <NS> apply -f kube/<NS>_configmap.yaml

4.3 After CI deploy — verify

kubectl -n <NS> get pods
kubectl -n <NS> rollout status deployment/<APP>-backend --timeout=180s

---

5. Common operations

• Rotate a secret: update the Gitea repo secret → push an empty commit or re-run workflow_dispatch → reconcile applies the new value → kubectl -n <NS> rollout restart deployment/<APP>-backend.
• If pods CrashLoopBackOff after a secret change: the env ref is likely unresolvable — check kubectl get event / secret keys; add the missing key to reconcile-cluster-inject.sh SECRET_KEYS and the workflow export block.
• Never delete the namespace on deploy (CI does not; don't start) — it destroys the live Secret.

---

6. Troubleshooting

• Build-deploy fails ~36s, "no remote endpoint provided": the buildx endpoint was put under driver-opts. Move it to a top-level with: input.
• Multi-arch build OOMs/crashes the buildkit pod: build linux/amd64 only.
• COPY --from=ghcr.io/... TLS timeout: buildkit can't reach ghcr.io. Use an ECR base + pip install uv.
• PR files API returns empty patch fields: build review payload from local git diff — never trust the API patch.

---

7. Handoff checklist

• All <PLACEHOLDERS> filled (workflow, reconcile script, kube/, this RUNBOOK)
• Repo secrets + vars set; keys match reconcile-cluster-inject.sh + workflow export block
• Branch protection on master enabled (push whitelist [armistace], merge whitelist [hermes, armistace])
• git config core.hooksPath ~/dev/git-hooks set in the new repo
• First deploy: namespace seeded with a real Secret (RUNBOOK §4.1) before relying on CI reconcile
• Any new env key added to §3.1 table + §2.1 list so a handoff doesn't lose it