Template
chore: seed gitea-repo-template cookie-cutter
Reusable repo template for homelab services on git.aridgwayweb.com. Bakes in (all verified live on armistace/wedding-photos): - Hard commit guard: shared pre-commit hook (git config core.hooksPath ~/dev/git-hooks) + master branch protection (push whitelist [armistace], merge whitelist [hermes, armistace]). - Gitea Actions CI (.gitea/workflows/build_push.yml): test + build-deploy, persistent remote buildkit cache, registry push, idempotent deploy that preserves hand-provisioned Secrets, cluster injection from repo secrets/vars via scripts/reconcile-cluster-inject.sh. - Persistent buildkit cache (ci/buildkit/): single-replica Longhorn backing. - scripts/reconcile-cluster-inject.sh: reconcile live Secret/ConfigMap from Gitea secrets/vars without clobbering hand-provisioned values. - RUNBOOK.md: handoff-complete ops doc. Placeholders (<APP> <OWNER> <NS> <KEY_*>) are filled per-service on repo creation.
This commit is contained in:
Hermes Agent
@@ -0,0 +1,27 @@
|
||||
# StorageClass for the persistent buildkit cache.
|
||||
# - numberOfReplicas: 1 -> the cache is written ONCE across the cluster (1x disk,
|
||||
# NOT the 3x triple-replication of the default 'longhorn' SC). This is deliberate:
|
||||
# a buildkit cache is throwable/recreatable, so replicating it 3x wastes disk that
|
||||
# node 2 (already 76%) can't afford. 1 replica survives single-node loss via
|
||||
# dataLocality: best-effort (re-replicates only when a node actually dies).
|
||||
# - dataLocality: best-effort -> keep the single replica on the same node as the
|
||||
# pod (fast local reads), replicate only if that node fails.
|
||||
# Auto-revert safety net: the buildkit-cache-monitor cron reverts buildkit to the
|
||||
# ephemeral (no-PVC) driver if any node crosses the critical disk threshold.
|
||||
# See monitoring/buildkit-cache-monitor.sh.
|
||||
apiVersion: storage.k8s.io/v1
|
||||
kind: StorageClass
|
||||
metadata:
|
||||
name: buildkit-single-1r
|
||||
annotations:
|
||||
description: "Single-replica Longhorn (buildkit cache) - 1x disk, durable across node loss"
|
||||
provisioner: driver.longhorn.io
|
||||
allowVolumeExpansion: true
|
||||
reclaimPolicy: Delete
|
||||
volumeBindingMode: Immediate
|
||||
parameters:
|
||||
numberOfReplicas: "1"
|
||||
staleReplicaTimeout: "30"
|
||||
fsType: "ext4"
|
||||
dataLocality: "best-effort"
|
||||
dataEngine: "v1"
|
||||
@@ -0,0 +1,77 @@
|
||||
# Dedicated long-lived buildkit daemon for CI image builds.
|
||||
#
|
||||
# Persists its build cache on a single-replica Longhorn volume (StorageClass
|
||||
# buildkit-single-1r) so repeated CI image builds reuse the layer cache instead of
|
||||
# re-pulling/building every time. The cache stays ONE copy on disk (not the 3x
|
||||
# default replication) and is bounded by the 10Gi PVC. The workflow's buildx uses
|
||||
# the REMOTE driver to point at this daemon via the ClusterIP Service below.
|
||||
#
|
||||
# Disk-pressure safety net: the buildkit-cache-monitor cron (monitoring/) reverts
|
||||
# CI to the ephemeral buildx driver if any node crosses critical disk usage.
|
||||
#
|
||||
# NOTE: no toleration for the gitea-builder taint on archlinux-k3s-1, so this pod
|
||||
# schedules on a worker node (2 or 3) away from the high-CPU control plane.
|
||||
apiVersion: apps/v1
|
||||
kind: StatefulSet
|
||||
metadata:
|
||||
name: buildkit
|
||||
namespace: gitea-runner
|
||||
spec:
|
||||
serviceName: buildkit
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app: buildkit
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: buildkit
|
||||
spec:
|
||||
containers:
|
||||
- name: buildkitd
|
||||
image: moby/buildkit:buildx-stable-1
|
||||
args:
|
||||
- --addr
|
||||
# Listen for the remote driver over TCP on 1234 (no TLS - internal cluster traffic).
|
||||
- tcp://0.0.0.0:1234
|
||||
# Keep the cache (do NOT use --oci-worker-no-process-sandbox or other
|
||||
# flags that break rootless cache persistence).
|
||||
ports:
|
||||
- name: daemon
|
||||
containerPort: 1234
|
||||
resources:
|
||||
requests:
|
||||
cpu: 500m
|
||||
memory: 1Gi
|
||||
limits:
|
||||
cpu: "2"
|
||||
memory: 4Gi
|
||||
ephemeral-storage: 2Gi
|
||||
volumeMounts:
|
||||
- name: cache
|
||||
mountPath: /var/lib/buildkit
|
||||
# BuildKit needs this for the CA/root store even when not using TLS.
|
||||
- name: certs
|
||||
mountPath: /etc/buildkit/certs
|
||||
- name: config
|
||||
mountPath: /etc/buildkit
|
||||
securityContext:
|
||||
# BuildKit's OCI/runc worker must bind-mount build contexts & layers.
|
||||
# The old driver-spawned (working) buildkit pods ran privileged:true;
|
||||
# false here caused "failed to mount snapshot ... operation not
|
||||
# permitted" (runc-native can't mount). Match the proven-working pods.
|
||||
privileged: true
|
||||
volumes:
|
||||
- name: certs
|
||||
emptyDir: {}
|
||||
- name: config
|
||||
emptyDir: {}
|
||||
volumeClaimTemplates:
|
||||
- metadata:
|
||||
name: cache
|
||||
spec:
|
||||
accessModes: ["ReadWriteOnce"]
|
||||
storageClassName: buildkit-single-1r
|
||||
resources:
|
||||
requests:
|
||||
storage: 10Gi
|
||||
@@ -0,0 +1,16 @@
|
||||
# ClusterIP service exposing the buildkit daemon to the CI runners.
|
||||
# The workflow's buildx remote driver connects to: tcp://buildkit.gitea-runner.svc:1234
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: buildkit
|
||||
namespace: gitea-runner
|
||||
spec:
|
||||
selector:
|
||||
app: buildkit
|
||||
ports:
|
||||
- name: daemon
|
||||
port: 1234
|
||||
targetPort: 1234
|
||||
protocol: TCP
|
||||
type: ClusterIP
|
||||
@@ -0,0 +1,26 @@
|
||||
# NetworkPolicy: only the Gitea action runners may reach the buildkit daemon.
|
||||
# Closes the reviewer's High: buildkit listens on plaintext tcp:1234 with no
|
||||
# auth — without this policy ANY cluster pod could submit arbitrary build
|
||||
# requests (lateral-movement / resource-abuse surface). Restricting ingress to
|
||||
# the gitea-runner namespace (where the runners live) removes that exposure.
|
||||
# Homelab note: full TLS+auth on the buildkit socket is deferred (documented) —
|
||||
# the runner and daemon are on the private cluster network; the policy closes the
|
||||
# pod-to-pod surface that TLS alone wouldn't.
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: buildkit-allow-runner-only
|
||||
namespace: gitea-runner
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app: buildkit
|
||||
policyTypes: ["Ingress"]
|
||||
ingress:
|
||||
- from:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: gitea-runner
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 1234
|
||||
@@ -0,0 +1,44 @@
|
||||
# Persistent Buildkit Cache for CI
|
||||
|
||||
Dedicated, long-lived buildkit daemon that persists its layer cache on a
|
||||
single-replica Longhorn PVC so repeated CI image builds reuse cache instead of
|
||||
re-building every time. This manifest set is **ported verbatim from
|
||||
`armistace/wedding-photos`** — verified live.
|
||||
|
||||
## Why single-replica Longhorn (not the default SC)
|
||||
|
||||
The buildx **kubernetes driver's PVC option creates the volume with NO
|
||||
`storageClassName`** — it falls back to the cluster default SC, which here is
|
||||
`longhorn` at **3 replicas**. That would triple every cache byte across
|
||||
disk-constrained nodes. So this uses a dedicated `buildkit-single-1r` SC
|
||||
(`numberOfReplicas: 1`, `dataLocality: best-effort`): the cache is **1× on
|
||||
disk** and only replicates if the hosting node actually dies.
|
||||
|
||||
## Components
|
||||
|
||||
| Path | Purpose |
|
||||
|---|---|
|
||||
| `ci/buildkit/01-storageclass.yaml` | `buildkit-single-1r` SC (1 replica, best-effort locality) |
|
||||
| `ci/buildkit/02-statefulset.yaml` | `buildkit` StatefulSet, pod schedules off the control plane, listens `tcp://0.0.0.0:1234`, 10Gi cache PVC |
|
||||
| `ci/buildkit/03-service.yaml` | ClusterIP `buildkit.gitea-runner.svc:1234` |
|
||||
| `ci/buildkit/04-networkpolicy.yaml` | restrict buildkit to the gitea-runner namespace only |
|
||||
| `.gitea/workflows/build_push.yml` | `setup-buildx-action` → `driver: remote`, `endpoint=tcp://buildkit.gitea-runner.svc:1234` |
|
||||
|
||||
> **Ops tooling is NOT in this repo.** The disk watchdog + revert/restore
|
||||
> scripts live on the control machine (`~/.hermes/scripts/buildkit-cache-monitor.sh`,
|
||||
> `buildkit-re-enable.sh`) and run from the `buildkit-disk-monitor` cron. They are
|
||||
> homelab ops, not app code.
|
||||
|
||||
## Apply (one-time, per cluster — NOT per repo)
|
||||
|
||||
```bash
|
||||
kubectl apply -f ci/buildkit/01-storageclass.yaml
|
||||
kubectl apply -f ci/buildkit/02-statefulset.yaml
|
||||
kubectl apply -f ci/buildkit/03-service.yaml
|
||||
kubectl apply -f ci/buildkit/04-networkpolicy.yaml
|
||||
kubectl -n gitea-runner rollout status statefulset/buildkit --timeout=180s
|
||||
```
|
||||
|
||||
Buildkit is a **cluster-wide shared resource** — it should be installed once, not
|
||||
once per repo. If your repo doesn't own the cluster, coordinate with whoever
|
||||
does (it lives in the `gitea-runner` namespace and only needs applying once).
|
||||
Reference in New Issue
Block a user