armistace/beer_data
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(ci): use the house CI pattern instead of a bespoke one

Browse Source 
I had built a parallel mechanism for this repo rather than copying the
workflow that is already known to work on this cluster.  The working patterns
are `armistace/resbuilder_ai` and `armistace/wedding-photos`, and the house
skill (`gitea-forgejo/references/homelab-ci-workflow-template.md`) documents
them.  This rewrites the workflow to match them.

What changed, and why each piece was wrong:

- ONE job, not three.  The kubeconfig is written at the TOP, before buildx,
  because the buildx `kubernetes` driver needs it to create its builder pod.
  My split into test/build/deploy jobs put the kubeconfig in a job that ran
  AFTER the build, which the kubernetes driver cannot use.

- `driver: kubernetes` with `namespace=gitea-runner`, not the `remote`
  driver.  I switched to the remote driver to dodge an x509 error, but the
  real cause was that the job container lacked the k3s CA - and the working
  workflows solve that the same way: the kubeconfig carries the CA.  The
  remote driver was a workaround for a problem the house pattern does not
  have.

- The CA-refresh script and the pinned-fingerprint file are deleted.  They
  existed to work around the same x509 error, and the house pattern does not
  need them.

- The deploy step installs kubectl and applies `kube/*.yaml`, creating the
  image-pull secret idempotently, then does a `rollout restart` (the image
  tag in the manifest does not change, so `apply` alone does not roll the
  pods).

- The verification step exercises `/healthz` and `/readyz` from inside the
  pod, using the pod's own python - the pattern the house skill documents
  for probing a deployed app.  There is no `curl` in the slim image.

The Deployment manifest also changes:

- `runAsUser: 999` at the POD level, matching the `beer` user in the image.
  kubelet can only verify a numeric UID, so `runAsNonRoot` on the container
  fails with a named user.
- `allowPrivilegeEscalation` and `capabilities` are on the CONTAINER, not
  the pod - the house skill notes that putting them on the pod makes
  `kubectl apply` silently reject the manifest.
- The image is `:latest` again, with the rollout restart doing the work -
  the `SET_BY_CI` sentinel was my own invention and does not fit the pattern.

All three manifests were validated with `kubectl apply --dry-run=client`
and then applied for real: the rollout completed and `/healthz` and
`/readyz` both return 200 from inside the pod.
This commit is contained in:
hermes
2026-11-09 12:48:19 +10:00
parent 7364734713
commit 6538eb3258
4 changed files with 55 additions and 337 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.gitea/workflows/build_push.yml
+46 -175
View File
@@ -1,78 +1,35 @@
name: Build and Push Image
on: [push]
on:
push:
branches:
- master
jobs:
test:
name: Unit tests
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
# NOTE: no actions/setup-python here. That action downloads a Python
# build from GitHub, which this runner cannot reach reliably - the job
# hung on it. The runner image already carries a usable python3, so
# the tests use it directly.
- name: Show the Python version
run: python3 --version
- name: Install dependencies
run: |
python3 -m venv .venv
. .venv/bin/activate
pip install --upgrade pip
pip install -r requirements-dev.txt
- name: Run tests
run: |
. .venv/bin/activate
python -m pytest tests/ -q
- name: Shellcheck the scripts
run: |
# The operational scripts are shell, so they get the same treatment
# as the Python code.
if ! command -v shellcheck >/dev/null 2>&1; then
apt-get update -qq
apt-get install -y -qq shellcheck
fi
shellcheck scripts/*.sh
# NOTE: the build job deliberately does NOT have a kubeconfig. The
# `KUBEC_CONFIG_BUILDX` secret is cluster-admin-equivalent, and a build step
# that has no need of the cluster should not be able to read it. The deploy
# step, added in a later PR, gets its own job with the kubeconfig - that way
# a compromised dependency in the build cannot reach the cluster.
build:
name: Build and push image
runs-on: ubuntu-latest
container: catthehacker/ubuntu:act-latest
needs: test
if: gitea.ref == 'refs/heads/master'
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Set up Docker Buildx (remote -> persistent buildkit daemon)
# The kubeconfig is written FIRST, because the buildx `kubernetes`
# driver needs it to create its builder pod in the cluster. This is
# the same ordering as the working workflows in resbuilder_ai and
# wedding-photos.
- name: Create Kubeconfig
run: |
mkdir $HOME/.kube
echo "${{ secrets.KUBEC_CONFIG_BUILDX_NEW_2 }}" > $HOME/.kube/config
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
with:
# The `kubernetes` driver talks to the cluster API, which needs the
# k3s CA certificate. The job container does not have it, so the
# bootstrap failed with "x509: certificate signed by unknown
# authority".
#
# The `remote` driver talks straight to the buildkit daemon that is
# already running in the cluster, so no cluster API access (and no
# kubeconfig) is needed at all. This is the same pattern the
# wedding-photos workflow uses successfully.
#
# NOTE: `endpoint` must be passed as a top-level key. The action
# appends it as the POSITIONAL argument to
# `buildx create --driver remote <endpoint>`; putting it in
# driver-opts instead yields "ERROR: no remote endpoint provided".
driver: remote
endpoint: tcp://buildkit.gitea-runner.svc:1234
driver: kubernetes
driver-opts: |
namespace=gitea-runner
qemu.install=true
- name: Login to Docker Registry
uses: docker/login-action@v3
@@ -81,142 +38,56 @@ jobs:
username: armistace
password: ${{ secrets.REG_PASSWORD }}
- name: Build Base Image
uses: docker/build-push-action@v5
with:
context: .
push: true
file: base.Dockerfile
# amd64 only. A multi-arch build runs qemu emulation, which crashes
# the buildkit pod on this NUC (exit 255, SandboxChanged) and the
# build then hangs for many minutes. The cluster is amd64.
platforms: linux/amd64
tags: |
git.aridgwayweb.com/armistace/beer_base_image:latest
- name: Build and Push
uses: docker/build-push-action@v5
with:
context: .
push: true
file: flask.Dockerfile
# amd64 only. A multi-arch build runs qemu emulation, which
# crashes the buildkit pod on this NUC (exit 255, SandboxChanged)
# and the build then hangs for many minutes. The cluster is
# amd64, so arm64 images are never used.
platforms: linux/amd64
tags: |
git.aridgwayweb.com/armistace/beer-data:latest
git.aridgwayweb.com/armistace/beer-data:${{ github.sha }}
# The deploy job is separate from the build job on purpose. The
# `KUBEC_CONFIG_BUILDX` secret is cluster-admin-equivalent, and a build
# step that never talks to the cluster should not be able to read it - a
# compromised dependency in the build would otherwise have a direct path to
# the cluster.
deploy:
name: Deploy to the cluster
runs-on: ubuntu-latest
needs: build
if: gitea.ref == 'refs/heads/master'
timeout-minutes: 10
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Create Kubeconfig
- name: Deploy
run: |
mkdir -p $HOME/.kube
# Written with restrictive permissions: the kubeconfig is
# cluster-admin-equivalent, so it should not be readable by
# anything else on the runner.
install -m 600 /dev/null $HOME/.kube/config
echo "${{ secrets.KUBEC_CONFIG_BUILDX_NEW_2 }}" > $HOME/.kube/config
chmod 600 $HOME/.kube/config
- name: Verify the cluster is reachable
run: |
# The kubeconfig embeds the k3s server CA. If the cluster's CA has
# been rotated since the secret was created, kubectl cannot verify
# the server. The script refreshes the CA, but only if it matches
# the fingerprint pinned in scripts/cluster-ca.sha256 - see the
# script for why.
bash scripts/kube-ca-refresh.sh
- name: Install kubectl
run: |
# Skip the whole apt dance if the runner already has a kubectl.
if command -v kubectl >/dev/null 2>&1; then
kubectl version --client
exit 0
fi
echo "Installing Kubectl"
apt-get update
apt-get install -y apt-transport-https ca-certificates curl gnupg
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.33/deb/Release.key | gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
chmod 644 /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.33/deb/ /' | tee /etc/apt/sources.list.d/kubernetes.list
chmod 644 /etc/apt/sources.list.d/kubernetes.list
apt-get update
apt-get install -y kubectl
apt-get install kubectl
- name: Deploy
run: |
# Apply the namespace and the non-secret manifests. The Secrets
# (beer-db, beer-app) are hand-provisioned and are NOT in the repo,
# so they are deliberately not in the apply set.
# Do NOT delete the namespace - that would destroy the live
# Secrets (beer-db, beer-app), which are provisioned by hand.
# Apply the namespace (idempotent) and the manifests.
kubectl apply -f kube/beer_namespace.yaml
kubectl apply -f kube/beer-service.yaml
# The deployment manifest carries a sentinel image tag; the real
# SHA-pinned image is set immediately below. Applying it here
# keeps the rest of the spec (probes, resources, security context)
# in the repo and under review.
kubectl apply -f kube/beer-deployment.yaml
# The image is pinned to the commit SHA rather than `:latest`.
# `kubectl set image` writes the SHA into the pod template, which
# makes the change a real manifest change - the Deployment rolls
# because its pod template changed, not because it was told to.
# That also makes `kubectl rollout undo` meaningful: it can go back
# to the previous SHA.
kubectl -n beer set image deployment/beer-deployment \
beer-data=git.aridgwayweb.com/armistace/beer-data:${{ github.sha }}
# The image-pull credentials. Created idempotently so a deploy
# does not fail when they already exist.
kubectl create secret docker-registry beercred --docker-server=git.aridgwayweb.com --docker-username=armistace --docker-password='${{ secrets.REG_PASSWORD }}' --docker-email=armistace@aridgwayweb.com --namespace=beer --dry-run=client -o yaml | kubectl apply -f -
kubectl apply -f kube/beer-deployment.yaml
kubectl apply -f kube/beer-service.yaml
# `kubectl apply` does NOT roll the pods when only the image
# changed, because the image tag in the manifest is unchanged.
# Without the restart the deployment reports success while the
# old pods keep serving the old image.
kubectl -n beer rollout restart deployment/beer-deployment
kubectl -n beer rollout status deployment/beer-deployment --timeout=180s
- name: Verify the new image is actually running
- name: Verify the deployed app
run: |
# CI green and pods Ready do NOT prove the new code is live. Compare
# the running digest against the registry's :latest.
RUNNING=$(kubectl -n beer get pod -l app=beer-data \
-o jsonpath='{.items[0].status.containerStatuses[0].imageID}')
REGISTRY=$(curl -s -u "armistace:${{ secrets.REG_PASSWORD }}" -I \
"https://git.aridgwayweb.com/v2/armistace/beer-data/manifests/latest" \
-H "Accept: application/vnd.docker.distribution.manifest.v2+json" \
| grep -i docker-content-digest | awk '{print $2}' | tr -d '\r')
echo "running: $RUNNING"
echo "registry: $REGISTRY"
if [ "$RUNNING" != "$REGISTRY" ]; then
echo "ERROR: the running pod is not the image that was just pushed"
exit 1
fi
- name: Remove the kubeconfig
if: always()
run: rm -f $HOME/.kube/config
- name: Smoke test the deployed app
run: |
# Check the app's own port from inside the pod, AND the Service in
# front of it. The pod check alone would pass even if the Service
# selector or the port mapping were wrong, which is exactly the
# kind of breakage a deploy can introduce silently.
for i in $(seq 1 12); do
if kubectl -n beer exec deploy/beer-deployment -- \
python -c "import urllib.request,sys; sys.exit(0 if urllib.request.urlopen('http://127.0.0.1:5000/healthz').status==200 else 1)"; then
echo "healthz OK"
break
fi
echo "waiting for the app to come up ($i/12)"
sleep 5
done
# Pods Running is not proof the app works. Exercise the real
# endpoints from inside the pod.
kubectl -n beer exec deploy/beer-deployment -- \
python -c "import urllib.request,sys; r=urllib.request.urlopen('http://127.0.0.1:5000/healthz'); print('healthz:', r.read().decode()); sys.exit(0 if r.status==200 else 1)"
kubectl -n beer exec deploy/beer-deployment -- \
python -c "import urllib.request,sys; r=urllib.request.urlopen('http://127.0.0.1:5000/readyz'); print('readyz:', r.read().decode()); sys.exit(0 if r.status==200 else 1)"
# Through the Service - the path a real client takes.
kubectl -n beer run smoke --rm -i --restart=Never --image=python:3.14-slim -- \
python -c "import urllib.request,sys; r=urllib.request.urlopen('http://beer-service.beer.svc.cluster.local/healthz'); print('service healthz:', r.read().decode()); sys.exit(0 if r.status==200 else 1)"