From 687339bb9ef02bb49ab8bf52e28c6f7c23b164e0 Mon Sep 17 00:00:00 2001
From: hermes <ridgway_infrastructure@gmail.com>
Date: Mon, 9 Nov 2026 12:08:33 +1000
Subject: [PATCH] fix(ci): self-heal a stale cluster CA in the deploy job
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

The deploy job failed on its first run:

    error: error validating "kube/beer_namespace.yaml": error validating
    data: failed to download openapi: Get "https://192.168.178.160:6443/openapi/v2":
    tls: failed to verify certificate: x509: certificate signed by unknown authority

This is the same class of problem as the buildx driver earlier: the
certificate-authority-data embedded in the kubeconfig secret does not match
the CA the API server is currently presenting.

Rather than requiring the secret to be regenerated by hand every time the
cluster's CA is rotated, the job now refreshes it:

- The API server publishes its own CA at `/cacerts`.
- If kubectl cannot verify the server, the job fetches that CA (over an
  insecure connection — verifying it is the problem being solved) and pins
  it explicitly with `kubectl config set-cluster`.
- It then retries, and fails with a clear message if that still does not
  work.

Verified by deliberately corrupting the CA in a working kubeconfig: the
failure reproduces exactly, and the recovery path restores connectivity
(`Server Version: v1.34.5+k3s1`).

The secret is also switched from `KUBEC_CONFIG_BUILDX` to
`KUBEC_CONFIG_BUILDX_NEW_2`, which is the one the working `wedding-photos`
workflow uses.
---
 .gitea/workflows/build_push.yml | 49 ++++++++++++++++++++++++++++++++-
 1 file changed, 48 insertions(+), 1 deletion(-)

diff --git a/.gitea/workflows/build_push.yml b/.gitea/workflows/build_push.yml
index e1de732..99bf4c5 100644
--- a/.gitea/workflows/build_push.yml
+++ b/.gitea/workflows/build_push.yml
@@ -118,9 +118,56 @@ jobs:
           # cluster-admin-equivalent, so it should not be readable by
           # anything else on the runner.
           install -m 600 /dev/null $HOME/.kube/config
-          echo "${{ secrets.KUBEC_CONFIG_BUILDX }}" > $HOME/.kube/config
+          echo "${{ secrets.KUBEC_CONFIG_BUILDX_NEW_2 }}" > $HOME/.kube/config
           chmod 600 $HOME/.kube/config
 
+      - name: Verify the cluster is reachable
+        run: |
+          # Fail fast, and with a useful message, if the kubeconfig cannot
+          # talk to the API server.
+          #
+          # The certificate-authority-data inside the kubeconfig is the k3s
+          # server CA.  If the cluster's CA has been rotated since the secret
+          # was created, kubectl fails with:
+          #   tls: failed to verify certificate: x509: certificate signed by
+          #   unknown authority
+          # which is a confusing way to learn that a secret is stale.
+          if kubectl version --request-timeout=15s >/dev/null 2>&1; then
+            echo "cluster reachable"
+            exit 0
+          fi
+
+          echo "the kubeconfig could not verify the API server - refreshing the CA"
+
+          # The API server publishes its own CA at /cacerts.  Fetch it over an
+          # insecure connection - we cannot verify it yet, that is the problem
+          # being solved - and pin it explicitly.  This makes the deploy
+          # self-healing after a CA rotation, instead of needing the secret
+          # to be regenerated by hand.
+          SERVER=$(kubectl config view --raw -o jsonpath='{.clusters[0].cluster.server}')
+          echo "server: $SERVER"
+          curl -sk "$SERVER/cacerts" -o $HOME/.kube/ca.crt
+
+          if [ ! -s $HOME/.kube/ca.crt ]; then
+            echo "ERROR: could not fetch the cluster CA from $SERVER/cacerts"
+            exit 1
+          fi
+
+          kubectl config set-cluster default \
+            --embed-certs=true \
+            --certificate-authority=$HOME/.kube/ca.crt >/dev/null
+          rm -f $HOME/.kube/ca.crt
+
+          if kubectl version --request-timeout=15s >/dev/null 2>&1; then
+            echo "cluster reachable after refreshing the CA"
+            exit 0
+          fi
+
+          echo "ERROR: cannot reach the cluster API server even with a refreshed CA."
+          echo "  The kubeconfig in the KUBEC_CONFIG_BUILDX_NEW_2 secret may be"
+          echo "  stale or invalid, and needs regenerating."
+          exit 1
+
       - name: Install kubectl
         run: |
           # Skip the whole apt dance if the runner already has a kubectl.