From 687339bb9ef02bb49ab8bf52e28c6f7c23b164e0 Mon Sep 17 00:00:00 2001 From: hermes Date: Mon, 9 Nov 2026 12:08:33 +1000 Subject: [PATCH 1/4] fix(ci): self-heal a stale cluster CA in the deploy job MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The deploy job failed on its first run: error: error validating "kube/beer_namespace.yaml": error validating data: failed to download openapi: Get "https://192.168.178.160:6443/openapi/v2": tls: failed to verify certificate: x509: certificate signed by unknown authority This is the same class of problem as the buildx driver earlier: the certificate-authority-data embedded in the kubeconfig secret does not match the CA the API server is currently presenting. Rather than requiring the secret to be regenerated by hand every time the cluster's CA is rotated, the job now refreshes it: - The API server publishes its own CA at `/cacerts`. - If kubectl cannot verify the server, the job fetches that CA (over an insecure connection — verifying it is the problem being solved) and pins it explicitly with `kubectl config set-cluster`. - It then retries, and fails with a clear message if that still does not work. Verified by deliberately corrupting the CA in a working kubeconfig: the failure reproduces exactly, and the recovery path restores connectivity (`Server Version: v1.34.5+k3s1`). The secret is also switched from `KUBEC_CONFIG_BUILDX` to `KUBEC_CONFIG_BUILDX_NEW_2`, which is the one the working `wedding-photos` workflow uses. --- .gitea/workflows/build_push.yml | 49 ++++++++++++++++++++++++++++++++- 1 file changed, 48 insertions(+), 1 deletion(-) diff --git a/.gitea/workflows/build_push.yml b/.gitea/workflows/build_push.yml index e1de732..99bf4c5 100644 --- a/.gitea/workflows/build_push.yml +++ b/.gitea/workflows/build_push.yml @@ -118,9 +118,56 @@ jobs: # cluster-admin-equivalent, so it should not be readable by # anything else on the runner. install -m 600 /dev/null $HOME/.kube/config - echo "${{ secrets.KUBEC_CONFIG_BUILDX }}" > $HOME/.kube/config + echo "${{ secrets.KUBEC_CONFIG_BUILDX_NEW_2 }}" > $HOME/.kube/config chmod 600 $HOME/.kube/config + - name: Verify the cluster is reachable + run: | + # Fail fast, and with a useful message, if the kubeconfig cannot + # talk to the API server. + # + # The certificate-authority-data inside the kubeconfig is the k3s + # server CA. If the cluster's CA has been rotated since the secret + # was created, kubectl fails with: + # tls: failed to verify certificate: x509: certificate signed by + # unknown authority + # which is a confusing way to learn that a secret is stale. + if kubectl version --request-timeout=15s >/dev/null 2>&1; then + echo "cluster reachable" + exit 0 + fi + + echo "the kubeconfig could not verify the API server - refreshing the CA" + + # The API server publishes its own CA at /cacerts. Fetch it over an + # insecure connection - we cannot verify it yet, that is the problem + # being solved - and pin it explicitly. This makes the deploy + # self-healing after a CA rotation, instead of needing the secret + # to be regenerated by hand. + SERVER=$(kubectl config view --raw -o jsonpath='{.clusters[0].cluster.server}') + echo "server: $SERVER" + curl -sk "$SERVER/cacerts" -o $HOME/.kube/ca.crt + + if [ ! -s $HOME/.kube/ca.crt ]; then + echo "ERROR: could not fetch the cluster CA from $SERVER/cacerts" + exit 1 + fi + + kubectl config set-cluster default \ + --embed-certs=true \ + --certificate-authority=$HOME/.kube/ca.crt >/dev/null + rm -f $HOME/.kube/ca.crt + + if kubectl version --request-timeout=15s >/dev/null 2>&1; then + echo "cluster reachable after refreshing the CA" + exit 0 + fi + + echo "ERROR: cannot reach the cluster API server even with a refreshed CA." + echo " The kubeconfig in the KUBEC_CONFIG_BUILDX_NEW_2 secret may be" + echo " stale or invalid, and needs regenerating." + exit 1 + - name: Install kubectl run: | # Skip the whole apt dance if the runner already has a kubectl. From ce6f5b7962c946be6b40531d74188af6b3285fe8 Mon Sep 17 00:00:00 2001 From: hermes Date: Mon, 9 Nov 2026 12:11:56 +1000 Subject: [PATCH 2/4] fix(ci): pin the cluster CA fingerprint instead of trusting a fetched CA The review was right on both counts, and both are fixed. 1. MITM. Fetching the CA over an unverified connection (`curl -sk`) and then trusting it is a privilege-escalation hole: the runner holds cluster-admin credentials, so an attacker able to intercept the fetch could hand us a CA of their choosing and then read everything sent over the connection. The CA is now PINNED. It is still fetched over an insecure connection, but it is only USED if its SHA-256 fingerprint matches `KUBE_SERVER_CA_FINGERPRINT`. An intercepted CA will not hash to the pinned value, so it is refused. If the secret is not set, the job fails rather than trusting an unverified CA. 2. The hardcoded cluster name. `kubectl config set-cluster default` creates a NEW cluster entry when the kubeconfig's entry is called something else, leaving the original (and its stale CA) in use. The name is now read from the kubeconfig: `{.clusters[0].name}`. The bash is also moved out of the YAML into `scripts/kube-ca-refresh.sh`, so it can be read, reviewed and tested on its own rather than as a wall of shell inside a workflow file. All three paths are tested against the live API server: - no fingerprint supplied -> refuses, exit 1 - wrong fingerprint -> refuses, exit 1 - correct fingerprint -> refreshes the CA, exit 0, kubectl then works --- .gitea/workflows/build_push.yml | 49 ++---------------- scripts/kube-ca-refresh.sh | 92 +++++++++++++++++++++++++++++++++ 2 files changed, 97 insertions(+), 44 deletions(-) create mode 100755 scripts/kube-ca-refresh.sh diff --git a/.gitea/workflows/build_push.yml b/.gitea/workflows/build_push.yml index 99bf4c5..4f46187 100644 --- a/.gitea/workflows/build_push.yml +++ b/.gitea/workflows/build_push.yml @@ -123,50 +123,11 @@ jobs: - name: Verify the cluster is reachable run: | - # Fail fast, and with a useful message, if the kubeconfig cannot - # talk to the API server. - # - # The certificate-authority-data inside the kubeconfig is the k3s - # server CA. If the cluster's CA has been rotated since the secret - # was created, kubectl fails with: - # tls: failed to verify certificate: x509: certificate signed by - # unknown authority - # which is a confusing way to learn that a secret is stale. - if kubectl version --request-timeout=15s >/dev/null 2>&1; then - echo "cluster reachable" - exit 0 - fi - - echo "the kubeconfig could not verify the API server - refreshing the CA" - - # The API server publishes its own CA at /cacerts. Fetch it over an - # insecure connection - we cannot verify it yet, that is the problem - # being solved - and pin it explicitly. This makes the deploy - # self-healing after a CA rotation, instead of needing the secret - # to be regenerated by hand. - SERVER=$(kubectl config view --raw -o jsonpath='{.clusters[0].cluster.server}') - echo "server: $SERVER" - curl -sk "$SERVER/cacerts" -o $HOME/.kube/ca.crt - - if [ ! -s $HOME/.kube/ca.crt ]; then - echo "ERROR: could not fetch the cluster CA from $SERVER/cacerts" - exit 1 - fi - - kubectl config set-cluster default \ - --embed-certs=true \ - --certificate-authority=$HOME/.kube/ca.crt >/dev/null - rm -f $HOME/.kube/ca.crt - - if kubectl version --request-timeout=15s >/dev/null 2>&1; then - echo "cluster reachable after refreshing the CA" - exit 0 - fi - - echo "ERROR: cannot reach the cluster API server even with a refreshed CA." - echo " The kubeconfig in the KUBEC_CONFIG_BUILDX_NEW_2 secret may be" - echo " stale or invalid, and needs regenerating." - exit 1 + # The kubeconfig embeds the k3s server CA. If the cluster's CA has + # been rotated since the secret was created, kubectl cannot verify + # the server. The script refreshes the CA, but only if it matches + # the fingerprint pinned in the secret - see the script for why. + bash scripts/kube-ca-refresh.sh "${{ secrets.KUBE_SERVER_CA_FINGERPRINT }}" - name: Install kubectl run: | diff --git a/scripts/kube-ca-refresh.sh b/scripts/kube-ca-refresh.sh new file mode 100755 index 0000000..f513919 --- /dev/null +++ b/scripts/kube-ca-refresh.sh @@ -0,0 +1,92 @@ +#!/usr/bin/env bash +# +# Make sure kubectl can talk to the cluster API server. +# +# The kubeconfig embeds the k3s server CA. If the cluster's CA has been +# rotated since the secret was created, kubectl fails with: +# +# tls: failed to verify certificate: x509: certificate signed by unknown authority +# +# which reads like a network problem rather than a stale secret. This script +# refreshes the CA in place. +# +# The refresh is PINNED: the server's CA is fetched over an unverified +# connection, but it is only used if its SHA-256 fingerprint matches the value +# in the KUBE_SERVER_CA_FINGERPRINT secret. An attacker able to intercept +# the fetch cannot use it - their CA will not hash to the pinned fingerprint. +# The runner has cluster-admin credentials, so trusting an unverified CA here +# would be a privilege-escalation hole. +# +# Usage: +# kube-ca-refresh.sh +# +# Exit codes: +# 0 the cluster is reachable +# 1 the cluster is not reachable and could not be fixed safely + +set -euo pipefail + +EXPECTED="${1:-}" + +if kubectl version --request-timeout=15s >/dev/null 2>&1; then + echo "cluster reachable" + exit 0 +fi + +if [ -z "$EXPECTED" ]; then + echo "ERROR: the kubeconfig cannot verify the API server, and no expected CA" >&2 + echo " fingerprint was supplied, so the CA cannot be refreshed safely." >&2 + echo " Set KUBE_SERVER_CA_FINGERPRINT to the SHA-256 fingerprint of the" >&2 + echo " cluster's CA:" >&2 + echo " openssl x509 -in server-ca.crt -noout -fingerprint -sha256" >&2 + exit 1 +fi + +SERVER=$(kubectl config view --raw -o jsonpath='{.clusters[0].cluster.server}') +# The cluster entry's NAME - patching the CA into a *new* entry named +# "default" would leave the original entry (and its stale CA) in use. +CLUSTER_NAME=$(kubectl config view --raw -o jsonpath='{.clusters[0].name}') +echo "server: $SERVER cluster entry: $CLUSTER_NAME" + +CA_FILE=$(mktemp) +trap 'rm -f "$CA_FILE"' EXIT + +curl -sk "$SERVER/cacerts" -o "$CA_FILE" +if [ ! -s "$CA_FILE" ]; then + echo "ERROR: could not fetch the cluster CA from $SERVER/cacerts" >&2 + exit 1 +fi + +# Normalise both to lowercase hex without separators, so the comparison does +# not depend on the colon-separated format openssl prints. +normalise() { + tr -d ':' | tr 'A-Z' 'a-z' +} + +ACTUAL=$(openssl x509 -in "$CA_FILE" -noout -fingerprint -sha256 | cut -d= -f2 | normalise) +WANTED=$(printf '%s' "$EXPECTED" | normalise) + +echo "expected CA fingerprint: $WANTED" +echo "actual CA fingerprint: $ACTUAL" + +if [ "$ACTUAL" != "$WANTED" ]; then + echo "ERROR: the cluster CA does not match the pinned fingerprint." >&2 + echo " Refusing to use it. Either the CA has been rotated and the" >&2 + echo " fingerprint secret needs updating, or the connection was" >&2 + echo " intercepted." >&2 + exit 1 +fi + +# The fingerprint matches the pinned value, so this is the CA we expect. +kubectl config set-cluster "$CLUSTER_NAME" \ + --embed-certs=true \ + --certificate-authority="$CA_FILE" >/dev/null + +if kubectl version --request-timeout=15s >/dev/null 2>&1; then + echo "cluster reachable after refreshing the CA" + exit 0 +fi + +echo "ERROR: cannot reach the cluster API server even with a refreshed CA." >&2 +echo " The kubeconfig secret may be stale or invalid." >&2 +exit 1 From a4b3d40e2d6686c5762c68a4ae53d46d16aab378 Mon Sep 17 00:00:00 2001 From: hermes Date: Mon, 9 Nov 2026 12:13:27 +1000 Subject: [PATCH 3/4] fix(ci): pin the cluster CA in the repo rather than in a secret The CA fingerprint is not a secret. It is a public value that anyone can read straight off the certificate, so pinning it in a Gitea secret was the wrong shape: - A change to it would be invisible - no commit, no review. - Nobody with secret-write access could silently repoint the CI runner at a different CA. It now lives in `scripts/cluster-ca.sha256`, so changing it is a reviewable commit. The script reads it from there, and still accepts an override as its first argument. This also removes the need for a `KUBE_SERVER_CA_FINGERPRINT` secret, which the bot token cannot create anyway - the Actions secrets API is owner-only. The behaviour is unchanged: the CA is fetched over an unverified connection but only USED if it matches the pinned fingerprint, so an intercepted CA is refused. Re-tested against the live API server - the matching fingerprint refreshes the CA and kubectl then works; a wrong one is refused and the stale CA is left in place. --- .gitea/workflows/build_push.yml | 5 +++-- scripts/cluster-ca.sha256 | 17 +++++++++++++++++ scripts/kube-ca-refresh.sh | 25 +++++++++++++++++++------ 3 files changed, 39 insertions(+), 8 deletions(-) create mode 100644 scripts/cluster-ca.sha256 diff --git a/.gitea/workflows/build_push.yml b/.gitea/workflows/build_push.yml index 4f46187..d013b01 100644 --- a/.gitea/workflows/build_push.yml +++ b/.gitea/workflows/build_push.yml @@ -126,8 +126,9 @@ jobs: # The kubeconfig embeds the k3s server CA. If the cluster's CA has # been rotated since the secret was created, kubectl cannot verify # the server. The script refreshes the CA, but only if it matches - # the fingerprint pinned in the secret - see the script for why. - bash scripts/kube-ca-refresh.sh "${{ secrets.KUBE_SERVER_CA_FINGERPRINT }}" + # the fingerprint pinned in scripts/cluster-ca.sha256 - see the + # script for why. + bash scripts/kube-ca-refresh.sh - name: Install kubectl run: | diff --git a/scripts/cluster-ca.sha256 b/scripts/cluster-ca.sha256 new file mode 100644 index 0000000..3f6664b --- /dev/null +++ b/scripts/cluster-ca.sha256 @@ -0,0 +1,17 @@ +# SHA-256 fingerprint of the cluster's API server CA (the k3s server-ca.crt). +# +# This is a PUBLIC value, not a secret - it can be read straight off the +# certificate. It lives in the repo rather than in a Gitea secret so that +# changing it is a reviewable commit, and so nobody with secret-write access +# can silently repoint the CI runner at a different CA. +# +# The deploy job fetches the cluster's CA over an unverified connection and +# compares its fingerprint against this value. A CA that does not match is +# refused, so an attacker able to intercept the fetch cannot substitute +# their own. +# +# To refresh it after a legitimate CA rotation: +# openssl x509 -in /var/lib/rancher/k3s/server/tls/server-ca.crt # -noout -fingerprint -sha256 +# and update the line below (colons are stripped before comparison, so either +# format works). +d23a17bb9c5f7c66d15e15635d6bb928e3a34f38142705e4e6516075a5f35907 diff --git a/scripts/kube-ca-refresh.sh b/scripts/kube-ca-refresh.sh index f513919..73a6b09 100755 --- a/scripts/kube-ca-refresh.sh +++ b/scripts/kube-ca-refresh.sh @@ -12,13 +12,21 @@ # # The refresh is PINNED: the server's CA is fetched over an unverified # connection, but it is only used if its SHA-256 fingerprint matches the value -# in the KUBE_SERVER_CA_FINGERPRINT secret. An attacker able to intercept -# the fetch cannot use it - their CA will not hash to the pinned fingerprint. +# pinned in scripts/cluster-ca.sha256. An attacker able to intercept the +# fetch cannot use it - their CA will not hash to the pinned fingerprint. # The runner has cluster-admin credentials, so trusting an unverified CA here # would be a privilege-escalation hole. # +# The fingerprint is a PUBLIC value - it can be read straight off the +# certificate - so it lives in the repo rather than in a secret. That way a +# change to it is a reviewable commit, and nobody with secret-write access +# can silently repoint the CI runner at a different CA. +# +# The expected fingerprint is read from scripts/cluster-ca.sha256 next to +# this script. It can be overridden with the first argument. +# # Usage: -# kube-ca-refresh.sh +# kube-ca-refresh.sh [expected-fingerprint] # # Exit codes: # 0 the cluster is reachable @@ -26,7 +34,12 @@ set -euo pipefail +SCRIPT_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) EXPECTED="${1:-}" +if [ -z "$EXPECTED" ] && [ -f "$SCRIPT_DIR/cluster-ca.sha256" ]; then + # The file may carry comments; take the last non-comment line. + EXPECTED=$(grep -vE '^[[:space:]]*(#|$)' "$SCRIPT_DIR/cluster-ca.sha256" | tail -1 | tr -d '[:space:]') +fi if kubectl version --request-timeout=15s >/dev/null 2>&1; then echo "cluster reachable" @@ -36,8 +49,8 @@ fi if [ -z "$EXPECTED" ]; then echo "ERROR: the kubeconfig cannot verify the API server, and no expected CA" >&2 echo " fingerprint was supplied, so the CA cannot be refreshed safely." >&2 - echo " Set KUBE_SERVER_CA_FINGERPRINT to the SHA-256 fingerprint of the" >&2 - echo " cluster's CA:" >&2 + echo " and no pinned CA fingerprint could be found." >&2 + echo " Set one in scripts/cluster-ca.sha256:" >&2 echo " openssl x509 -in server-ca.crt -noout -fingerprint -sha256" >&2 exit 1 fi @@ -72,7 +85,7 @@ echo "actual CA fingerprint: $ACTUAL" if [ "$ACTUAL" != "$WANTED" ]; then echo "ERROR: the cluster CA does not match the pinned fingerprint." >&2 echo " Refusing to use it. Either the CA has been rotated and the" >&2 - echo " fingerprint secret needs updating, or the connection was" >&2 + echo " pinned fingerprint needs updating, or the connection was" >&2 echo " intercepted." >&2 exit 1 fi From 3e20b558a57fec9fea53b17abbc0cc71da50212c Mon Sep 17 00:00:00 2001 From: hermes Date: Mon, 9 Nov 2026 12:15:50 +1000 Subject: [PATCH 4/4] fix(ci): harden the CA refresh script and add shellcheck to CI Round-2 review findings, all low severity. - `curl` now has `--connect-timeout` and `--max-time`, so an unreachable API server fails the job instead of hanging the runner indefinitely. - A guard for a kubeconfig with no cluster entries, or an empty one, with a clear message rather than an opaque kubectl error. - The fingerprint mismatch message now prints both the expected and the actual value, which cuts the time to diagnose the next CA rotation. - `set -euo pipefail` was already there; the script now passes shellcheck clean, and a shellcheck step was added to the CI test job so the operational scripts get the same treatment as the Python code. --- .gitea/workflows/build_push.yml | 10 ++++++++++ scripts/kube-ca-refresh.sh | 23 +++++++++++++++++++++-- 2 files changed, 31 insertions(+), 2 deletions(-) diff --git a/.gitea/workflows/build_push.yml b/.gitea/workflows/build_push.yml index d013b01..8adc926 100644 --- a/.gitea/workflows/build_push.yml +++ b/.gitea/workflows/build_push.yml @@ -28,6 +28,16 @@ jobs: . .venv/bin/activate python -m pytest tests/ -q + - name: Shellcheck the scripts + run: | + # The operational scripts are shell, so they get the same treatment + # as the Python code. + if ! command -v shellcheck >/dev/null 2>&1; then + apt-get update -qq + apt-get install -y -qq shellcheck + fi + shellcheck scripts/*.sh + # NOTE: the build job deliberately does NOT have a kubeconfig. The # `KUBEC_CONFIG_BUILDX` secret is cluster-admin-equivalent, and a build step # that has no need of the cluster should not be able to read it. The deploy diff --git a/scripts/kube-ca-refresh.sh b/scripts/kube-ca-refresh.sh index 73a6b09..f80117b 100755 --- a/scripts/kube-ca-refresh.sh +++ b/scripts/kube-ca-refresh.sh @@ -55,16 +55,33 @@ if [ -z "$EXPECTED" ]; then exit 1 fi +# Guard against a kubeconfig with no clusters, or an empty one. +CLUSTER_COUNT=$(kubectl config view --raw -o jsonpath='{.clusters}' 2>/dev/null \ + | grep -c '"name"' || true) +if [ "${CLUSTER_COUNT:-0}" -eq 0 ]; then + echo "ERROR: the kubeconfig has no cluster entries." >&2 + echo " The KUBEC_CONFIG_BUILDX_NEW_2 secret looks wrong or empty." >&2 + exit 1 +fi + SERVER=$(kubectl config view --raw -o jsonpath='{.clusters[0].cluster.server}') # The cluster entry's NAME - patching the CA into a *new* entry named # "default" would leave the original entry (and its stale CA) in use. CLUSTER_NAME=$(kubectl config view --raw -o jsonpath='{.clusters[0].name}') + +if [ -z "$SERVER" ] || [ -z "$CLUSTER_NAME" ]; then + echo "ERROR: could not read the server or cluster name from the kubeconfig." >&2 + exit 1 +fi + echo "server: $SERVER cluster entry: $CLUSTER_NAME" CA_FILE=$(mktemp) trap 'rm -f "$CA_FILE"' EXIT -curl -sk "$SERVER/cacerts" -o "$CA_FILE" +# Timeouts so an unreachable API server fails the job rather than hanging +# the runner indefinitely. +curl -sk --connect-timeout 10 --max-time 30 "$SERVER/cacerts" -o "$CA_FILE" if [ ! -s "$CA_FILE" ]; then echo "ERROR: could not fetch the cluster CA from $SERVER/cacerts" >&2 exit 1 @@ -73,7 +90,7 @@ fi # Normalise both to lowercase hex without separators, so the comparison does # not depend on the colon-separated format openssl prints. normalise() { - tr -d ':' | tr 'A-Z' 'a-z' + tr -d ':' | tr '[:upper:]' '[:lower:]' } ACTUAL=$(openssl x509 -in "$CA_FILE" -noout -fingerprint -sha256 | cut -d= -f2 | normalise) @@ -84,6 +101,8 @@ echo "actual CA fingerprint: $ACTUAL" if [ "$ACTUAL" != "$WANTED" ]; then echo "ERROR: the cluster CA does not match the pinned fingerprint." >&2 + echo " expected: $WANTED" >&2 + echo " actual: $ACTUAL" >&2 echo " Refusing to use it. Either the CA has been rotated and the" >&2 echo " pinned fingerprint needs updating, or the connection was" >&2 echo " intercepted." >&2