The review was right on both counts, and both are fixed.
1. MITM. Fetching the CA over an unverified connection (`curl -sk`) and then
trusting it is a privilege-escalation hole: the runner holds cluster-admin
credentials, so an attacker able to intercept the fetch could hand us a CA
of their choosing and then read everything sent over the connection.
The CA is now PINNED. It is still fetched over an insecure connection, but
it is only USED if its SHA-256 fingerprint matches
`KUBE_SERVER_CA_FINGERPRINT`. An intercepted CA will not hash to the
pinned value, so it is refused. If the secret is not set, the job fails
rather than trusting an unverified CA.
2. The hardcoded cluster name. `kubectl config set-cluster default` creates a
NEW cluster entry when the kubeconfig's entry is called something else,
leaving the original (and its stale CA) in use. The name is now read from
the kubeconfig: `{.clusters[0].name}`.
The bash is also moved out of the YAML into `scripts/kube-ca-refresh.sh`, so
it can be read, reviewed and tested on its own rather than as a wall of
shell inside a workflow file.
All three paths are tested against the live API server:
- no fingerprint supplied -> refuses, exit 1
- wrong fingerprint -> refuses, exit 1
- correct fingerprint -> refreshes the CA, exit 0, kubectl then works
212 lines
9.2 KiB
YAML
212 lines
9.2 KiB
YAML
name: Build and Push Image
|
|
on: [push]
|
|
|
|
jobs:
|
|
test:
|
|
name: Unit tests
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- name: Checkout
|
|
uses: actions/checkout@v4
|
|
|
|
# NOTE: no actions/setup-python here. That action downloads a Python
|
|
# build from GitHub, which this runner cannot reach reliably - the job
|
|
# hung on it. The runner image already carries a usable python3, so
|
|
# the tests use it directly.
|
|
- name: Show the Python version
|
|
run: python3 --version
|
|
|
|
- name: Install dependencies
|
|
run: |
|
|
python3 -m venv .venv
|
|
. .venv/bin/activate
|
|
pip install --upgrade pip
|
|
pip install -r requirements-dev.txt
|
|
|
|
- name: Run tests
|
|
run: |
|
|
. .venv/bin/activate
|
|
python -m pytest tests/ -q
|
|
|
|
# NOTE: the build job deliberately does NOT have a kubeconfig. The
|
|
# `KUBEC_CONFIG_BUILDX` secret is cluster-admin-equivalent, and a build step
|
|
# that has no need of the cluster should not be able to read it. The deploy
|
|
# step, added in a later PR, gets its own job with the kubeconfig - that way
|
|
# a compromised dependency in the build cannot reach the cluster.
|
|
build:
|
|
name: Build and push image
|
|
runs-on: ubuntu-latest
|
|
container: catthehacker/ubuntu:act-latest
|
|
needs: test
|
|
if: gitea.ref == 'refs/heads/master'
|
|
|
|
steps:
|
|
- name: Checkout
|
|
uses: actions/checkout@v4
|
|
|
|
- name: Set up Docker Buildx (remote -> persistent buildkit daemon)
|
|
uses: docker/setup-buildx-action@v3
|
|
with:
|
|
# The `kubernetes` driver talks to the cluster API, which needs the
|
|
# k3s CA certificate. The job container does not have it, so the
|
|
# bootstrap failed with "x509: certificate signed by unknown
|
|
# authority".
|
|
#
|
|
# The `remote` driver talks straight to the buildkit daemon that is
|
|
# already running in the cluster, so no cluster API access (and no
|
|
# kubeconfig) is needed at all. This is the same pattern the
|
|
# wedding-photos workflow uses successfully.
|
|
#
|
|
# NOTE: `endpoint` must be passed as a top-level key. The action
|
|
# appends it as the POSITIONAL argument to
|
|
# `buildx create --driver remote <endpoint>`; putting it in
|
|
# driver-opts instead yields "ERROR: no remote endpoint provided".
|
|
driver: remote
|
|
endpoint: tcp://buildkit.gitea-runner.svc:1234
|
|
|
|
- name: Login to Docker Registry
|
|
uses: docker/login-action@v3
|
|
with:
|
|
registry: git.aridgwayweb.com
|
|
username: armistace
|
|
password: ${{ secrets.REG_PASSWORD }}
|
|
|
|
- name: Build Base Image
|
|
uses: docker/build-push-action@v5
|
|
with:
|
|
context: .
|
|
push: true
|
|
file: base.Dockerfile
|
|
# amd64 only. A multi-arch build runs qemu emulation, which crashes
|
|
# the buildkit pod on this NUC (exit 255, SandboxChanged) and the
|
|
# build then hangs for many minutes. The cluster is amd64.
|
|
platforms: linux/amd64
|
|
tags: |
|
|
git.aridgwayweb.com/armistace/beer_base_image:latest
|
|
|
|
- name: Build and Push
|
|
uses: docker/build-push-action@v5
|
|
with:
|
|
context: .
|
|
push: true
|
|
file: flask.Dockerfile
|
|
platforms: linux/amd64
|
|
tags: |
|
|
git.aridgwayweb.com/armistace/beer-data:latest
|
|
git.aridgwayweb.com/armistace/beer-data:${{ github.sha }}
|
|
|
|
# The deploy job is separate from the build job on purpose. The
|
|
# `KUBEC_CONFIG_BUILDX` secret is cluster-admin-equivalent, and a build
|
|
# step that never talks to the cluster should not be able to read it - a
|
|
# compromised dependency in the build would otherwise have a direct path to
|
|
# the cluster.
|
|
deploy:
|
|
name: Deploy to the cluster
|
|
runs-on: ubuntu-latest
|
|
needs: build
|
|
if: gitea.ref == 'refs/heads/master'
|
|
timeout-minutes: 10
|
|
|
|
steps:
|
|
- name: Checkout
|
|
uses: actions/checkout@v4
|
|
|
|
- name: Create Kubeconfig
|
|
run: |
|
|
mkdir -p $HOME/.kube
|
|
# Written with restrictive permissions: the kubeconfig is
|
|
# cluster-admin-equivalent, so it should not be readable by
|
|
# anything else on the runner.
|
|
install -m 600 /dev/null $HOME/.kube/config
|
|
echo "${{ secrets.KUBEC_CONFIG_BUILDX_NEW_2 }}" > $HOME/.kube/config
|
|
chmod 600 $HOME/.kube/config
|
|
|
|
- name: Verify the cluster is reachable
|
|
run: |
|
|
# The kubeconfig embeds the k3s server CA. If the cluster's CA has
|
|
# been rotated since the secret was created, kubectl cannot verify
|
|
# the server. The script refreshes the CA, but only if it matches
|
|
# the fingerprint pinned in the secret - see the script for why.
|
|
bash scripts/kube-ca-refresh.sh "${{ secrets.KUBE_SERVER_CA_FINGERPRINT }}"
|
|
|
|
- name: Install kubectl
|
|
run: |
|
|
# Skip the whole apt dance if the runner already has a kubectl.
|
|
if command -v kubectl >/dev/null 2>&1; then
|
|
kubectl version --client
|
|
exit 0
|
|
fi
|
|
apt-get update
|
|
apt-get install -y apt-transport-https ca-certificates curl gnupg
|
|
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.33/deb/Release.key | gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
|
|
chmod 644 /etc/apt/keyrings/kubernetes-apt-keyring.gpg
|
|
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.33/deb/ /' | tee /etc/apt/sources.list.d/kubernetes.list
|
|
apt-get update
|
|
apt-get install -y kubectl
|
|
|
|
- name: Deploy
|
|
run: |
|
|
# Apply the namespace and the non-secret manifests. The Secrets
|
|
# (beer-db, beer-app) are hand-provisioned and are NOT in the repo,
|
|
# so they are deliberately not in the apply set.
|
|
kubectl apply -f kube/beer_namespace.yaml
|
|
kubectl apply -f kube/beer-service.yaml
|
|
# The deployment manifest carries a sentinel image tag; the real
|
|
# SHA-pinned image is set immediately below. Applying it here
|
|
# keeps the rest of the spec (probes, resources, security context)
|
|
# in the repo and under review.
|
|
kubectl apply -f kube/beer-deployment.yaml
|
|
|
|
# The image is pinned to the commit SHA rather than `:latest`.
|
|
# `kubectl set image` writes the SHA into the pod template, which
|
|
# makes the change a real manifest change - the Deployment rolls
|
|
# because its pod template changed, not because it was told to.
|
|
# That also makes `kubectl rollout undo` meaningful: it can go back
|
|
# to the previous SHA.
|
|
kubectl -n beer set image deployment/beer-deployment \
|
|
beer-data=git.aridgwayweb.com/armistace/beer-data:${{ github.sha }}
|
|
kubectl -n beer rollout status deployment/beer-deployment --timeout=180s
|
|
|
|
- name: Verify the new image is actually running
|
|
run: |
|
|
# CI green and pods Ready do NOT prove the new code is live. Compare
|
|
# the running digest against the registry's :latest.
|
|
RUNNING=$(kubectl -n beer get pod -l app=beer-data \
|
|
-o jsonpath='{.items[0].status.containerStatuses[0].imageID}')
|
|
REGISTRY=$(curl -s -u "armistace:${{ secrets.REG_PASSWORD }}" -I \
|
|
"https://git.aridgwayweb.com/v2/armistace/beer-data/manifests/latest" \
|
|
-H "Accept: application/vnd.docker.distribution.manifest.v2+json" \
|
|
| grep -i docker-content-digest | awk '{print $2}' | tr -d '\r')
|
|
echo "running: $RUNNING"
|
|
echo "registry: $REGISTRY"
|
|
if [ "$RUNNING" != "$REGISTRY" ]; then
|
|
echo "ERROR: the running pod is not the image that was just pushed"
|
|
exit 1
|
|
fi
|
|
|
|
- name: Remove the kubeconfig
|
|
if: always()
|
|
run: rm -f $HOME/.kube/config
|
|
|
|
- name: Smoke test the deployed app
|
|
run: |
|
|
# Check the app's own port from inside the pod, AND the Service in
|
|
# front of it. The pod check alone would pass even if the Service
|
|
# selector or the port mapping were wrong, which is exactly the
|
|
# kind of breakage a deploy can introduce silently.
|
|
for i in $(seq 1 12); do
|
|
if kubectl -n beer exec deploy/beer-deployment -- \
|
|
python -c "import urllib.request,sys; sys.exit(0 if urllib.request.urlopen('http://127.0.0.1:5000/healthz').status==200 else 1)"; then
|
|
echo "healthz OK"
|
|
break
|
|
fi
|
|
echo "waiting for the app to come up ($i/12)"
|
|
sleep 5
|
|
done
|
|
kubectl -n beer exec deploy/beer-deployment -- \
|
|
python -c "import urllib.request,sys; r=urllib.request.urlopen('http://127.0.0.1:5000/readyz'); print('readyz:', r.read().decode()); sys.exit(0 if r.status==200 else 1)"
|
|
|
|
# Through the Service - the path a real client takes.
|
|
kubectl -n beer run smoke --rm -i --restart=Never --image=python:3.14-slim -- \
|
|
python -c "import urllib.request,sys; r=urllib.request.urlopen('http://beer-service.beer.svc.cluster.local/healthz'); print('service healthz:', r.read().decode()); sys.exit(0 if r.status==200 else 1)"
|