I had built a parallel mechanism for this repo rather than copying the
workflow that is already known to work on this cluster. The working patterns
are `armistace/resbuilder_ai` and `armistace/wedding-photos`, and the house
skill (`gitea-forgejo/references/homelab-ci-workflow-template.md`) documents
them. This rewrites the workflow to match them.
What changed, and why each piece was wrong:
- ONE job, not three. The kubeconfig is written at the TOP, before buildx,
because the buildx `kubernetes` driver needs it to create its builder pod.
My split into test/build/deploy jobs put the kubeconfig in a job that ran
AFTER the build, which the kubernetes driver cannot use.
- `driver: kubernetes` with `namespace=gitea-runner`, not the `remote`
driver. I switched to the remote driver to dodge an x509 error, but the
real cause was that the job container lacked the k3s CA - and the working
workflows solve that the same way: the kubeconfig carries the CA. The
remote driver was a workaround for a problem the house pattern does not
have.
- The CA-refresh script and the pinned-fingerprint file are deleted. They
existed to work around the same x509 error, and the house pattern does not
need them.
- The deploy step installs kubectl and applies `kube/*.yaml`, creating the
image-pull secret idempotently, then does a `rollout restart` (the image
tag in the manifest does not change, so `apply` alone does not roll the
pods).
- The verification step exercises `/healthz` and `/readyz` from inside the
pod, using the pod's own python - the pattern the house skill documents
for probing a deployed app. There is no `curl` in the slim image.
The Deployment manifest also changes:
- `runAsUser: 999` at the POD level, matching the `beer` user in the image.
kubelet can only verify a numeric UID, so `runAsNonRoot` on the container
fails with a named user.
- `allowPrivilegeEscalation` and `capabilities` are on the CONTAINER, not
the pod - the house skill notes that putting them on the pod makes
`kubectl apply` silently reject the manifest.
- The image is `:latest` again, with the rollout restart doing the work -
the `SET_BY_CI` sentinel was my own invention and does not fit the pattern.
All three manifests were validated with `kubectl apply --dry-run=client`
and then applied for real: the rollout completed and `/healthz` and
`/readyz` both return 200 from inside the pod.
The review flagged `:latest` as a critical issue, and it is right: a mutable
tag makes `kubectl rollout undo` meaningless and lets different pods in the
same deployment run different code if they restart at different times.
- The build job now pushes the image with BOTH `:latest` and
`:${{ github.sha }}`.
- The deployment manifest carries a sentinel tag rather than `:latest`, and
the deploy job sets the real SHA with `kubectl set image`. That makes the
deploy a genuine manifest change, so the Deployment rolls because its pod
template changed — not because it was told to. `kubectl rollout undo` now
means something: it goes back to the previous SHA.
- The digest check now compares against the SHA tag rather than `:latest`,
and it checks EVERY running pod instead of `items[0]`. During a rolling
update `items[0]` may still be an old pod that has not been terminated,
which would compare against the wrong digest. The job also fails if the
deployment is not fully rolled out.
- The smoke test now checks the app through the Service as well as directly
in the pod. The pod check alone would pass even if the Service selector
or the port mapping were wrong.
- `readOnlyRootFilesystem: true` — the app writes no files, so the root
filesystem can be read-only. /tmp is still writable via the emptyDir.
- The kubectl install is skipped when the runner already has one.
The cluster objects for the beer namespace were created by hand. This puts
them in the repo so they can be applied and reviewed like everything else.
- `beer-deployment.yaml` — the Deployment, with the security context, the
resource limits, and both probes wired up (`/healthz` liveness,
`/readyz` readiness).
- `beer-service.yaml` — the NodePort service.
- `beer_namespace.yaml` — the namespace.
The credentials are NOT in these files. `MONGO_PASS` and `SECRET_KEY` come
from the live `beer-db` and `beer-app` Secrets, so applying the manifests
cannot clobber a hand-provisioned credential.
Notes on the pod security context
---------------------------------
`runAsNonRoot: true` cannot be set on the container when the image's user is
a *named* user - kubelet can only verify a numeric UID. The pod-level
security context therefore sets `runAsUser: 999`, which matches the `beer`
user the image creates (verified with `id` in the built image).
The container also drops all capabilities and refuses privilege escalation.
hermes