beer_data/RUNBOOK.md

RUNBOOK — Brew Log (beer_data)

Operational notes for the beer namespace deployment.

What runs where

Namespace	beer
Deployment	beer-deployment (3 replicas)
Image	git.aridgwayweb.com/armistace/beer-data:latest
Service	beer-service, NodePort 30015 → container port 5000
Database	MongoDB at 192.168.178.229, database beer_db
Secret	beercred (registry pull credentials)

Access: http://<any-cluster-node>:30015/

The pods are pinned to archlinux-k3s-3 (.161) via nodeSelector.

Environment variables

Variable	Value	Source
MONGO_HOST	192.168.178.229	Deployment manifest
MONGO_USER	root	Deployment manifest
MONGO_PASS	(see below)	Deployment manifest
SECRET_KEY	(see below)	Deployment manifest — Flask session signing key. Required
FLASK_ENV	production	Dockerfile
FLASK_DEBUG	0	Dockerfile

SECRET_KEY and MONGO_PASS must be set in the live Deployment. They are not committed to this repo — read them from the running cluster:

kubectl -n beer get deploy beer-deployment -o jsonpath='{.spec.template.spec.containers[0].env}'

The app refuses to start without a real SECRET_KEY. There is no default and no placeholder: a missing or placeholder value raises ImproperlyConfigured and the container exits. A known session-signing key would let anyone forge a login cookie, so a misconfigured deployment must fail loudly rather than run insecurely.

Generate one with:

python -c 'import secrets; print(secrets.token_hex(32))'

Moving the credentials into a Kubernetes Secret

The live Deployment currently carries MONGO_PASS as a plain environment value. Moving it to a Secret is the better shape and is not done here because it would mean replacing the live hand-provisioned values — a change that needs the operator's sign-off. When it is done:

env:
  - name: MONGO_PASS
    valueFrom:
      secretKeyRef:
        name: beer-db
        key: mongo_pass
  - name: SECRET_KEY
    valueFrom:
      secretKeyRef:
        name: beer-app
        key: secret_key

Deploying

Pushing to master triggers .gitea/workflows/build_push.yml, which builds the image, pushes it, deploys it to the cluster and verifies it. There is no manual step.

The workflow is a single job, in this order:

1. Checkout
2. Create Kubeconfig — writes secrets.KUBEC_CONFIG_BUILDX_NEW_2 to $HOME/.kube/config. This is FIRST because the buildx kubernetes driver needs it to create its builder pod in the cluster.
3. Set up Docker Buildx — driver: kubernetes, namespace=gitea-runner.
4. Login to the registry — secrets.REG_PASSWORD.
5. Build and Push — linux/amd64 only. A multi-arch build runs qemu, which crashes the buildkit pod on this NUC (exit 255, SandboxChanged) and the build then hangs for many minutes.
6. Deploy — installs kubectl, applies kube/*.yaml, creates the image-pull secret idempotently, then rollout restart + rollout status. The rollout restart is required: the image tag in the manifest is :latest, so kubectl apply alone does not roll the pods.
7. Verify the deployed app — exercises /healthz and /readyz from inside the pod, using the pod's own python (there is no curl in the slim image).

The deploy step does not delete the namespace. That would destroy the hand-provisioned beer-db and beer-app Secrets.

Deploying by hand

If you need to deploy without going through CI — for example to test a manifest change — do the same thing the workflow does:

kubectl apply -f kube/beer_namespace.yaml
kubectl apply -f kube/beer-deployment.yaml
kubectl apply -f kube/beer-service.yaml
kubectl -n beer rollout restart deployment/beer-deployment
kubectl -n beer rollout status deployment/beer-deployment --timeout=180s

Validate the manifests before applying them. kubectl apply silently rejects a manifest with a schema error, so a broken manifest leaves the old Deployment in place while the job still reports success:

kubectl apply --dry-run=client -f kube/

Verify the deploy actually took

CI green and pods Ready do not prove the new code is running. Check the running digest against the registry:

# what the pod is running
kubectl -n beer get pod -l app=beer-data \
  -o jsonpath='{.items[0].status.containerStatuses[0].imageID}'

# what the registry has as :latest
curl -s -u "armistace:$REG_PASSWORD" -I \
  "https://git.aridgwayweb.com/v2/armistace/beer-data/manifests/latest" \
  -H "Accept: application/vnd.docker.distribution.manifest.v2+json" \
  | grep -i docker-content-digest

If they differ, the registry :latest is stale — re-run the build rather than re-applying the manifest.

Check the app is alive

kubectl -n beer get pods
kubectl -n beer logs deploy/beer-deployment --tail=50
curl -s http://<node>:30015/healthz     # -> {"status":"ok"}
curl -s http://<node>:30015/readyz      # -> {"status":"ready"}

/healthz is the liveness probe and never touches Mongo — it answers "is the process up". A database outage must not make the kubelet restart a healthy process.

/readyz is the readiness probe and pings Mongo, returning 503 when the database is unreachable. Wire it to the Service so a pod with no database is pulled out of the endpoints rather than serving errors:

readinessProbe:
  httpGet:
    path: /readyz
    port: 5000
  initialDelaySeconds: 5
  periodSeconds: 10
livenessProbe:
  httpGet:
    path: /healthz
    port: 5000
  initialDelaySeconds: 10
  periodSeconds: 20

The run metadata collection

The beer_runs collection holds one document per brew run with the details that cannot be derived from the readings: the canonical name, the tap number, the recorded original gravity and the brew notes.

It starts empty on a fresh install. Populate it from the existing readings:

# from the app's source tree, with the usual env vars set
python scripts/backfill_run_metadata.py            # dry run
python scripts/backfill_run_metadata.py --apply     # write it

The script is idempotent and only ever inserts into beer_runs. It does not touch beer_data.

To see what is in it:

mongosh "mongodb://root:$MONGO_PASS@192.168.178.229:27017/beer_db" \
  --eval 'db.beer_runs.find().pretty()'

Data safety

The beer_db.beer_data collection holds the actual brewing records. There is no migration step and no schema version — documents are read as they are written, and every field is treated as optional.

To take a backup before any risky change:

mongodump --uri="mongodb://root:$MONGO_PASS@192.168.178.229:27017/beer_db" \
  --out=/tmp/beer_db_backup_$(date +%Y%m%d)

Common problems

The login page loads but nothing else does

The Mongo connection failed. Check the app logs for an auth error and confirm the credentials:

kubectl -n beer logs deploy/beer-deployment --tail=100 | grep -i mongo

A run shows a negative alcohol figure

That was the old formula ((max - min) / 7.36 + 0.5) applied to a run whose current gravity is above the assumed original gravity of 1012. Fixed in the current version: the standard (OG - FG) × 131.25 formula is used, and a run with no known OG shows a dash instead of a figure.

The pod will not start after a deploy

The most likely cause is a missing SECRET_KEY — the app exits rather than run with a default. Check the logs:

kubectl -n beer logs deploy/beer-deployment --tail=20
# ImproperlyConfigured: SECRET_KEY is not set. ...

The chart on the run page is blank

The chart library is served from /static/js/chart.umd.js — it is vendored in the repo, not loaded from a CDN, so the container works with no outbound network. Check the browser console for a 404 on that path.