fix(ci): use the house CI pattern instead of a bespoke one
I had built a parallel mechanism for this repo rather than copying the workflow that is already known to work on this cluster. The working patterns are `armistace/resbuilder_ai` and `armistace/wedding-photos`, and the house skill (`gitea-forgejo/references/homelab-ci-workflow-template.md`) documents them. This rewrites the workflow to match them. What changed, and why each piece was wrong: - ONE job, not three. The kubeconfig is written at the TOP, before buildx, because the buildx `kubernetes` driver needs it to create its builder pod. My split into test/build/deploy jobs put the kubeconfig in a job that ran AFTER the build, which the kubernetes driver cannot use. - `driver: kubernetes` with `namespace=gitea-runner`, not the `remote` driver. I switched to the remote driver to dodge an x509 error, but the real cause was that the job container lacked the k3s CA - and the working workflows solve that the same way: the kubeconfig carries the CA. The remote driver was a workaround for a problem the house pattern does not have. - The CA-refresh script and the pinned-fingerprint file are deleted. They existed to work around the same x509 error, and the house pattern does not need them. - The deploy step installs kubectl and applies `kube/*.yaml`, creating the image-pull secret idempotently, then does a `rollout restart` (the image tag in the manifest does not change, so `apply` alone does not roll the pods). - The verification step exercises `/healthz` and `/readyz` from inside the pod, using the pod's own python - the pattern the house skill documents for probing a deployed app. There is no `curl` in the slim image. The Deployment manifest also changes: - `runAsUser: 999` at the POD level, matching the `beer` user in the image. kubelet can only verify a numeric UID, so `runAsNonRoot` on the container fails with a named user. - `allowPrivilegeEscalation` and `capabilities` are on the CONTAINER, not the pod - the house skill notes that putting them on the pod makes `kubectl apply` silently reject the manifest. - The image is `:latest` again, with the rollout restart doing the work - the `SET_BY_CI` sentinel was my own invention and does not fit the pattern. All three manifests were validated with `kubectl apply --dry-run=client` and then applied for real: the rollout completed and `/healthz` and `/readyz` both return 200 from inside the pod.
This commit is contained in:
@@ -1,17 +0,0 @@
|
||||
# SHA-256 fingerprint of the cluster's API server CA (the k3s server-ca.crt).
|
||||
#
|
||||
# This is a PUBLIC value, not a secret - it can be read straight off the
|
||||
# certificate. It lives in the repo rather than in a Gitea secret so that
|
||||
# changing it is a reviewable commit, and so nobody with secret-write access
|
||||
# can silently repoint the CI runner at a different CA.
|
||||
#
|
||||
# The deploy job fetches the cluster's CA over an unverified connection and
|
||||
# compares its fingerprint against this value. A CA that does not match is
|
||||
# refused, so an attacker able to intercept the fetch cannot substitute
|
||||
# their own.
|
||||
#
|
||||
# To refresh it after a legitimate CA rotation:
|
||||
# openssl x509 -in /var/lib/rancher/k3s/server/tls/server-ca.crt # -noout -fingerprint -sha256
|
||||
# and update the line below (colons are stripped before comparison, so either
|
||||
# format works).
|
||||
d23a17bb9c5f7c66d15e15635d6bb928e3a34f38142705e4e6516075a5f35907
|
||||
@@ -1,126 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
#
|
||||
# Make sure kubectl can talk to the cluster API server.
|
||||
#
|
||||
# The kubeconfig embeds the k3s server CA. If the cluster's CA has been
|
||||
# rotated since the secret was created, kubectl fails with:
|
||||
#
|
||||
# tls: failed to verify certificate: x509: certificate signed by unknown authority
|
||||
#
|
||||
# which reads like a network problem rather than a stale secret. This script
|
||||
# refreshes the CA in place.
|
||||
#
|
||||
# The refresh is PINNED: the server's CA is fetched over an unverified
|
||||
# connection, but it is only used if its SHA-256 fingerprint matches the value
|
||||
# pinned in scripts/cluster-ca.sha256. An attacker able to intercept the
|
||||
# fetch cannot use it - their CA will not hash to the pinned fingerprint.
|
||||
# The runner has cluster-admin credentials, so trusting an unverified CA here
|
||||
# would be a privilege-escalation hole.
|
||||
#
|
||||
# The fingerprint is a PUBLIC value - it can be read straight off the
|
||||
# certificate - so it lives in the repo rather than in a secret. That way a
|
||||
# change to it is a reviewable commit, and nobody with secret-write access
|
||||
# can silently repoint the CI runner at a different CA.
|
||||
#
|
||||
# The expected fingerprint is read from scripts/cluster-ca.sha256 next to
|
||||
# this script. It can be overridden with the first argument.
|
||||
#
|
||||
# Usage:
|
||||
# kube-ca-refresh.sh [expected-fingerprint]
|
||||
#
|
||||
# Exit codes:
|
||||
# 0 the cluster is reachable
|
||||
# 1 the cluster is not reachable and could not be fixed safely
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
|
||||
EXPECTED="${1:-}"
|
||||
if [ -z "$EXPECTED" ] && [ -f "$SCRIPT_DIR/cluster-ca.sha256" ]; then
|
||||
# The file may carry comments; take the last non-comment line.
|
||||
EXPECTED=$(grep -vE '^[[:space:]]*(#|$)' "$SCRIPT_DIR/cluster-ca.sha256" | tail -1 | tr -d '[:space:]')
|
||||
fi
|
||||
|
||||
if kubectl version --request-timeout=15s >/dev/null 2>&1; then
|
||||
echo "cluster reachable"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if [ -z "$EXPECTED" ]; then
|
||||
echo "ERROR: the kubeconfig cannot verify the API server, and no expected CA" >&2
|
||||
echo " fingerprint was supplied, so the CA cannot be refreshed safely." >&2
|
||||
echo " and no pinned CA fingerprint could be found." >&2
|
||||
echo " Set one in scripts/cluster-ca.sha256:" >&2
|
||||
echo " openssl x509 -in server-ca.crt -noout -fingerprint -sha256" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Guard against a kubeconfig with no clusters, or an empty one.
|
||||
# `grep -c` counts LINES, and the jsonpath output is a single line, so the
|
||||
# count is done with grep -o and wc -l instead.
|
||||
CLUSTER_COUNT=$(kubectl config view --raw -o jsonpath='{.clusters[*].name}' 2>/dev/null \
|
||||
| grep -o . | wc -l || true)
|
||||
if [ "${CLUSTER_COUNT:-0}" -eq 0 ]; then
|
||||
echo "ERROR: the kubeconfig has no cluster entries." >&2
|
||||
echo " The KUBEC_CONFIG_BUILDX_NEW_2 secret looks wrong or empty." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
SERVER=$(kubectl config view --raw -o jsonpath='{.clusters[0].cluster.server}')
|
||||
# The cluster entry's NAME - patching the CA into a *new* entry named
|
||||
# "default" would leave the original entry (and its stale CA) in use.
|
||||
CLUSTER_NAME=$(kubectl config view --raw -o jsonpath='{.clusters[0].name}')
|
||||
|
||||
if [ -z "$SERVER" ] || [ -z "$CLUSTER_NAME" ]; then
|
||||
echo "ERROR: could not read the server or cluster name from the kubeconfig." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "server: $SERVER cluster entry: $CLUSTER_NAME"
|
||||
|
||||
CA_FILE=$(mktemp)
|
||||
trap 'rm -f "$CA_FILE"' EXIT
|
||||
|
||||
# Timeouts so an unreachable API server fails the job rather than hanging
|
||||
# the runner indefinitely.
|
||||
curl -sk --connect-timeout 10 --max-time 30 "$SERVER/cacerts" -o "$CA_FILE"
|
||||
if [ ! -s "$CA_FILE" ]; then
|
||||
echo "ERROR: could not fetch the cluster CA from $SERVER/cacerts" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Normalise both to lowercase hex without separators, so the comparison does
|
||||
# not depend on the colon-separated format openssl prints.
|
||||
normalise() {
|
||||
tr -d ':' | tr '[:upper:]' '[:lower:]'
|
||||
}
|
||||
|
||||
ACTUAL=$(openssl x509 -in "$CA_FILE" -noout -fingerprint -sha256 | cut -d= -f2 | normalise)
|
||||
WANTED=$(printf '%s' "$EXPECTED" | normalise)
|
||||
|
||||
echo "expected CA fingerprint: $WANTED"
|
||||
echo "actual CA fingerprint: $ACTUAL"
|
||||
|
||||
if [ "$ACTUAL" != "$WANTED" ]; then
|
||||
echo "ERROR: the cluster CA does not match the pinned fingerprint." >&2
|
||||
echo " expected: $WANTED" >&2
|
||||
echo " actual: $ACTUAL" >&2
|
||||
echo " Refusing to use it. Either the CA has been rotated and the" >&2
|
||||
echo " pinned fingerprint needs updating, or the connection was" >&2
|
||||
echo " intercepted." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# The fingerprint matches the pinned value, so this is the CA we expect.
|
||||
kubectl config set-cluster "$CLUSTER_NAME" \
|
||||
--embed-certs=true \
|
||||
--certificate-authority="$CA_FILE" >/dev/null
|
||||
|
||||
if kubectl version --request-timeout=15s >/dev/null 2>&1; then
|
||||
echo "cluster reachable after refreshing the CA"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
echo "ERROR: cannot reach the cluster API server even with a refreshed CA." >&2
|
||||
echo " The kubeconfig secret may be stale or invalid." >&2
|
||||
exit 1
|
||||
Reference in New Issue
Block a user