hermes
4d1e070d74
The RUNBOOK still said the workflow did not deploy to the cluster. It does now - it builds, pushes, applies the manifests, rolls the pods and verifies the app from inside the pod. The manual deploy path is documented alongside it, along with the dry-run step that catches the schema errors kubectl apply silently swallows.
241 lines
7.9 KiB
Markdown
241 lines
7.9 KiB
Markdown
# RUNBOOK — Brew Log (beer_data)
|
||
|
||
Operational notes for the `beer` namespace deployment.
|
||
|
||
## What runs where
|
||
|
||
| | |
|
||
|---|---|
|
||
| Namespace | `beer` |
|
||
| Deployment | `beer-deployment` (3 replicas) |
|
||
| Image | `git.aridgwayweb.com/armistace/beer-data:latest` |
|
||
| Service | `beer-service`, NodePort **30015** → container port 5000 |
|
||
| Database | MongoDB at `192.168.178.229`, database `beer_db` |
|
||
| Secret | `beercred` (registry pull credentials) |
|
||
|
||
Access: `http://<any-cluster-node>:30015/`
|
||
|
||
The pods are pinned to `archlinux-k3s-3` (`.161`) via `nodeSelector`.
|
||
|
||
## Environment variables
|
||
|
||
| Variable | Value | Source |
|
||
|---|---|---|
|
||
| `MONGO_HOST` | `192.168.178.229` | Deployment manifest |
|
||
| `MONGO_USER` | `root` | Deployment manifest |
|
||
| `MONGO_PASS` | *(see below)* | Deployment manifest |
|
||
| `SECRET_KEY` | *(see below)* | Deployment manifest — Flask session signing key. **Required** |
|
||
| `FLASK_ENV` | `production` | Dockerfile |
|
||
| `FLASK_DEBUG` | `0` | Dockerfile |
|
||
|
||
`SECRET_KEY` and `MONGO_PASS` must be set in the live Deployment. They are
|
||
**not** committed to this repo — read them from the running cluster:
|
||
|
||
```bash
|
||
kubectl -n beer get deploy beer-deployment -o jsonpath='{.spec.template.spec.containers[0].env}'
|
||
```
|
||
|
||
> **The app refuses to start without a real `SECRET_KEY`.** There is no default
|
||
> and no placeholder: a missing or placeholder value raises
|
||
> `ImproperlyConfigured` and the container exits. A known session-signing key
|
||
> would let anyone forge a login cookie, so a misconfigured deployment must
|
||
> fail loudly rather than run insecurely.
|
||
>
|
||
> Generate one with:
|
||
> ```bash
|
||
> python -c 'import secrets; print(secrets.token_hex(32))'
|
||
> ```
|
||
|
||
### Moving the credentials into a Kubernetes Secret
|
||
|
||
The live Deployment currently carries `MONGO_PASS` as a plain environment
|
||
value. Moving it to a Secret is the better shape and is **not** done here
|
||
because it would mean replacing the live hand-provisioned values — a change
|
||
that needs the operator's sign-off. When it is done:
|
||
|
||
```yaml
|
||
env:
|
||
- name: MONGO_PASS
|
||
valueFrom:
|
||
secretKeyRef:
|
||
name: beer-db
|
||
key: mongo_pass
|
||
- name: SECRET_KEY
|
||
valueFrom:
|
||
secretKeyRef:
|
||
name: beer-app
|
||
key: secret_key
|
||
```
|
||
|
||
## Deploying
|
||
|
||
Pushing to `master` triggers `.gitea/workflows/build_push.yml`, which builds
|
||
the image, pushes it, **deploys it to the cluster and verifies it**. There is
|
||
no manual step.
|
||
|
||
The workflow is a single job, in this order:
|
||
|
||
1. **Checkout**
|
||
2. **Create Kubeconfig** — writes `secrets.KUBEC_CONFIG_BUILDX_NEW_2` to
|
||
`$HOME/.kube/config`. This is FIRST because the buildx `kubernetes` driver
|
||
needs it to create its builder pod in the cluster.
|
||
3. **Set up Docker Buildx** — `driver: kubernetes`, `namespace=gitea-runner`.
|
||
4. **Login to the registry** — `secrets.REG_PASSWORD`.
|
||
5. **Build and Push** — `linux/amd64` only. A multi-arch build runs qemu,
|
||
which crashes the buildkit pod on this NUC (exit 255, `SandboxChanged`)
|
||
and the build then hangs for many minutes.
|
||
6. **Deploy** — installs kubectl, applies `kube/*.yaml`, creates the
|
||
image-pull secret idempotently, then `rollout restart` + `rollout status`.
|
||
The rollout restart is required: the image tag in the manifest is
|
||
`:latest`, so `kubectl apply` alone does not roll the pods.
|
||
7. **Verify the deployed app** — exercises `/healthz` and `/readyz` from
|
||
inside the pod, using the pod's own python (there is no `curl` in the
|
||
slim image).
|
||
|
||
> The deploy step does **not** delete the namespace. That would destroy the
|
||
> hand-provisioned `beer-db` and `beer-app` Secrets.
|
||
|
||
### Deploying by hand
|
||
|
||
If you need to deploy without going through CI — for example to test a
|
||
manifest change — do the same thing the workflow does:
|
||
|
||
```bash
|
||
kubectl apply -f kube/beer_namespace.yaml
|
||
kubectl apply -f kube/beer-deployment.yaml
|
||
kubectl apply -f kube/beer-service.yaml
|
||
kubectl -n beer rollout restart deployment/beer-deployment
|
||
kubectl -n beer rollout status deployment/beer-deployment --timeout=180s
|
||
```
|
||
|
||
Validate the manifests before applying them. `kubectl apply` **silently
|
||
rejects** a manifest with a schema error, so a broken manifest leaves the
|
||
old Deployment in place while the job still reports success:
|
||
|
||
```bash
|
||
kubectl apply --dry-run=client -f kube/
|
||
```
|
||
|
||
### Verify the deploy actually took
|
||
|
||
CI green and pods `Ready` do **not** prove the new code is running. Check the
|
||
running digest against the registry:
|
||
|
||
```bash
|
||
# what the pod is running
|
||
kubectl -n beer get pod -l app=beer-data \
|
||
-o jsonpath='{.items[0].status.containerStatuses[0].imageID}'
|
||
|
||
# what the registry has as :latest
|
||
curl -s -u "armistace:$REG_PASSWORD" -I \
|
||
"https://git.aridgwayweb.com/v2/armistace/beer-data/manifests/latest" \
|
||
-H "Accept: application/vnd.docker.distribution.manifest.v2+json" \
|
||
| grep -i docker-content-digest
|
||
```
|
||
|
||
If they differ, the registry `:latest` is stale — re-run the build rather than
|
||
re-applying the manifest.
|
||
|
||
### Check the app is alive
|
||
|
||
```bash
|
||
kubectl -n beer get pods
|
||
kubectl -n beer logs deploy/beer-deployment --tail=50
|
||
curl -s http://<node>:30015/healthz # -> {"status":"ok"}
|
||
curl -s http://<node>:30015/readyz # -> {"status":"ready"}
|
||
```
|
||
|
||
`/healthz` is the **liveness** probe and never touches Mongo — it answers "is
|
||
the process up". A database outage must not make the kubelet restart a healthy
|
||
process.
|
||
|
||
`/readyz` is the **readiness** probe and pings Mongo, returning 503 when the
|
||
database is unreachable. Wire it to the Service so a pod with no database is
|
||
pulled out of the endpoints rather than serving errors:
|
||
|
||
```yaml
|
||
readinessProbe:
|
||
httpGet:
|
||
path: /readyz
|
||
port: 5000
|
||
initialDelaySeconds: 5
|
||
periodSeconds: 10
|
||
livenessProbe:
|
||
httpGet:
|
||
path: /healthz
|
||
port: 5000
|
||
initialDelaySeconds: 10
|
||
periodSeconds: 20
|
||
```
|
||
|
||
## The run metadata collection
|
||
|
||
The `beer_runs` collection holds one document per brew run with the details
|
||
that cannot be derived from the readings: the canonical name, the tap number,
|
||
the recorded original gravity and the brew notes.
|
||
|
||
It starts empty on a fresh install. Populate it from the existing readings:
|
||
|
||
```bash
|
||
# from the app's source tree, with the usual env vars set
|
||
python scripts/backfill_run_metadata.py # dry run
|
||
python scripts/backfill_run_metadata.py --apply # write it
|
||
```
|
||
|
||
The script is idempotent and only ever inserts into `beer_runs`. It does not
|
||
touch `beer_data`.
|
||
|
||
To see what is in it:
|
||
|
||
```bash
|
||
mongosh "mongodb://root:$MONGO_PASS@192.168.178.229:27017/beer_db" \
|
||
--eval 'db.beer_runs.find().pretty()'
|
||
```
|
||
|
||
## Data safety
|
||
|
||
The `beer_db.beer_data` collection holds the actual brewing records. There is
|
||
no migration step and no schema version — documents are read as they are
|
||
written, and every field is treated as optional.
|
||
|
||
To take a backup before any risky change:
|
||
|
||
```bash
|
||
mongodump --uri="mongodb://root:$MONGO_PASS@192.168.178.229:27017/beer_db" \
|
||
--out=/tmp/beer_db_backup_$(date +%Y%m%d)
|
||
```
|
||
|
||
## Common problems
|
||
|
||
### The login page loads but nothing else does
|
||
|
||
The Mongo connection failed. Check the app logs for an auth error and confirm
|
||
the credentials:
|
||
|
||
```bash
|
||
kubectl -n beer logs deploy/beer-deployment --tail=100 | grep -i mongo
|
||
```
|
||
|
||
### A run shows a negative alcohol figure
|
||
|
||
That was the old formula (`(max - min) / 7.36 + 0.5`) applied to a run whose
|
||
current gravity is above the assumed original gravity of 1012. Fixed in the
|
||
current version: the standard `(OG - FG) × 131.25` formula is used, and a run
|
||
with no known OG shows a dash instead of a figure.
|
||
|
||
### The pod will not start after a deploy
|
||
|
||
The most likely cause is a missing `SECRET_KEY` — the app exits rather than
|
||
run with a default. Check the logs:
|
||
|
||
```bash
|
||
kubectl -n beer logs deploy/beer-deployment --tail=20
|
||
# ImproperlyConfigured: SECRET_KEY is not set. ...
|
||
```
|
||
|
||
### The chart on the run page is blank
|
||
|
||
The chart library is served from `/static/js/chart.umd.js` — it is vendored in
|
||
the repo, not loaded from a CDN, so the container works with no outbound
|
||
network. Check the browser console for a 404 on that path.
|