homelab

Talos Kubernetes on a Proxmox cluster. Pulumi (Python) provisions VMs + Talos; Helmfile installs the base cluster stack and apps; Ansible handles auxiliary (non-Talos) hosts. State stored in Garage S3 on TrueNAS.

Layout

pulumi/    Pulumi (Python) — Talos VMs on Proxmox + PKI + bootstrap + kubeconfig
talos/     Local kubeconfig/talosconfig mirror + workflow docs
k8s/       Helmfile + local charts for the base cluster stack and apps
ansible/   Playbooks/roles for auxiliary (non-Talos) hosts
docs/      Architecture and runbooks

Prerequisites

• devenv (loads .env automatically, provides all tools via Nix)
• Proxmox API access — dedicated pulumi@pve user with PulumiRole ACL and Datastore.AllocateSpace on local storage
• Garage S3 bucket pulumi on TrueNAS for Pulumi state
• Talos nocloud ISO from factory.talos.dev uploaded to Proxmox iso storage
• TLS certs for every ingress hostname staged on the TrueNAS NFS share at /mnt/ALEXANDRIA/certs (see TLS certs)

First-time setup

1. .env

Copy .env.example → .env and fill in real values:

# Garage S3 — Pulumi state
AWS_ACCESS_KEY_ID=<garage-key-id>
AWS_SECRET_ACCESS_KEY=<garage-secret>
AWS_REGION=garage
PULUMI_BACKEND_URL=s3://pulumi?region=garage&endpoint=http://nas.lan.xbazzi.com:30188&s3ForcePathStyle=true&disableSSL=true
PULUMI_CONFIG_PASSPHRASE=<passphrase>

# Proxmox API token (pulumi@pve!homelab=<uuid>)
PROXMOX_VE_API_TOKEN=<token>
PROXMOX_VE_SSH_USERNAME=root

2. Enter the devenv shell

direnv allow   # or: devenv shell

3. SOPS + helm plugins (one-time)

sops:init

This generates .age-key.txt (gitignored) and installs the helm-secrets + helm-diff plugins. It prints the public key — paste it into .sops.yaml replacing REPLACE_ME_WITH_YOUR_AGE_PUBLIC_KEY.

Back up .age-key.txt to 1Password / Bitwarden / paper. Losing it means losing every SOPS-encrypted secret in the repo permanently. (No SOPS-encrypted files ship in the base stack today; kept around for future secrets.)

4. Pulumi login + stack init

pulumi login $PULUMI_BACKEND_URL   # first time only
cd pulumi
pip install -r requirements.txt    # installs into .venv
pulumi stack init dev              # first time only

Bring up the cluster

cluster:up

Runs pulumi (VMs + PKI + etcd bootstrap + kubeconfig) → talos:sync-config (writes kubeconfig/talosconfig to disk) → cluster:apply (helmfile applies Cilium, LB pool, Traefik, csi-driver-nfs). Safe to re-run: pulumi diffs, helmfile is idempotent.

For a full teardown + rebuild:

redeploy    # = cluster:down (drain PVCs, destroy VMs; NAS subdirs preserved) then cluster:up

Deploy the higher layers

The in-cluster stack is split across four helmfiles so each layer's cadence is independent:

cluster:apply    # base:apply + db:apply + metrics:apply (three layers, dep-ordered)
apps:deploy      # user apps (whoami, ...)

Or apply layers individually:

base:apply       # base infra (Cilium, CoreDNS, LB, Traefik+dashboard, csi-driver-nfs, portainer-agent)
db:apply         # local-path + CNPG operator + shared pg-cluster (PG_MODE=recovery|standalone)
metrics:apply    # metrics-server + kube-prometheus-stack + Loki + Alloy

cluster:up runs pulumi:up + talos:sync-config + cluster:apply in one shot. Apps are opt-in follow-up. Each helmfile is idempotent; safe to re-run.

Apps live in k8s/helmfile-apps.yaml — one release per user app, each backed by a local chart in k8s/charts/.

CI-driven apps (coaching)

Every release here is declaratively pinned except one. The coaching app (source: ~/repos/coaching, mirrored to gitgud.boo/xbazzi/coaching) is built and deployed by Forgejo Actions on every push to master:

push master
  └─ .forgejo/workflows/deploy.yml   (Swarm runner, docker.sock)
       ├─ docker build -f deploy/Dockerfile
       ├─ push gitgud.boo/xbazzi/coaching:{sha-<short>,master}
       └─ kubectl set image deploy/coaching app=…:sha-<short> migrate=…:sha-<short>
            as ServiceAccount coaching-deployer — RBAC-scoped to patching
            that one Deployment, nothing else in the cluster

The pod's migrate initContainer runs alembic upgrade head before the app starts, so a schema change ships with the code that needs it. A failed migration leaves the previous ReplicaSet serving and fails the CI job.

The job runs on a self-hosted runner, and must. runs-on: homelab targets forrest_gump, a Forgejo Runner v12 on the Swarm host prod2 (/docker-shared/stacks/compose/forgejo-runner/). gitgud.boo is someone else's infrastructure and its shared runners — which also answer ubuntu-latest — have no route to 10.67.0.0/22, so the rollout step there dies with dial tcp 10.67.0.10:6443: i/o timeout. No credential or DNS change fixes that; only running on an in-network host does.

Consequence of pinning: if that runner is down the job queues instead of failing. A job stuck "waiting" means check the runner first — sudo docker service logs forgejo-runner_runner on prod2.

If the rollout step ever does fail, the image is already pushed but nothing deployed it: imagePullPolicy: Always only applies when a pod is created, so the running pod keeps serving the old code. Recover with:

coaching:rollout

Note apps:deploy will NOT do this — helmfile only restarts a pod when the rendered manifest changes, and a new image behind the same :master tag changes nothing. rollout restart is what forces the re-pull.

The runner itself

Not in this repo — it lives on prod2 at /docker-shared/stacks/compose/forgejo-runner/{docker-compose.yml,config.yaml}, which is an NFS share with no version control. Redeploy with sudo docker stack deploy -c docker-compose.yml forgejo-runner from that dir.

Non-obvious bits, all of which were failure modes first:

Registration	Forgejo Runner v12 uses declarative server.connections (uuid + secret). The register subcommand is deprecated and rejects a UI-issued runner credential with the misleading runner registration token not found. A UUID is 32 hex chars; the secret is 40.
Secret	Swarm secret forgejo_runner_token, consumed via token_url: file:/run/secrets/.... TODO: rotate — it was exposed in a chat transcript.
docker_host	Must be "automount". In v12 both "" and "-" mean no Docker in the job container, unlike older act_runner configs where "" meant auto.
Command	The image has no ENTRYPOINT (Cmd is ["/bin/forgejo-runner"]), so compose command: must name the binary explicitly.
User	user: "1000:995" — uid the image runs as, gid of prod2's docker group, needed for the mounted socket. Not group_add, which docker stack deploy silently ignores.
State dir	/var/lib/forgejo-runner on local ext4, deliberately not /docker-shared: that NFS export squashes writes to root:root, and git then refuses its own action cache with detected dubious ownership.
Label	Advertises homelab only. It must never claim ubuntu-latest — the other repos use that and this runner holds the host Docker socket (root-equivalent on prod2).

The Dockerfile must stay classic-builder compatible (no RUN --mount, no # syntax=): the job image's docker CLI ships no buildx plugin, and CLI 23+ dropped the daemon-side BuildKit path, so DOCKER_BUILDKIT=1 fails outright.

Tag drift is expected. The chart declares image.tag: master; CI deploys the immutable :sha-<short>. A later apps:deploy re-pins the pod spec to :master, which resolves to the same digest — a tag rename, not a rollback. To actually roll back:

kubectl -n coaching set image deploy/coaching \
  app=gitgud.boo/xbazzi/coaching:sha-<old> \
  migrate=gitgud.boo/xbazzi/coaching:sha-<old>

Two one-time setup steps, both outside this repo:

1. REGISTRY_TOKEN — Forgejo repo secret, write:package scope. A second, read-only token goes in k8s/base-infra/coaching/secrets.yaml (SOPS) and becomes the kubelet's pull credential. Keep them separate.

2. KUBE_CONFIG — Forgejo repo secret, base64 of a kubeconfig for the coaching-deployer ServiceAccount:

   TOKEN=$(kubectl -n coaching get secret coaching-deployer-token \
             -o jsonpath='{.data.token}' | base64 -d)
   CA=$(kubectl -n coaching get secret coaching-deployer-token \
             -o jsonpath='{.data.ca\.crt}')
   # assemble a kubeconfig with server https://10.67.0.10:6443,
   # certificate-authority-data $CA, token $TOKEN — then base64 the whole file.
   

   Address the API server by IP, not k8s.lan.xbazzi.com. The Forgejo job container resolves via Docker's embedded DNS (127.0.0.11), which forwards to the daemon's upstream resolvers — and since the Swarm host runs systemd-resolved on 127.0.0.53, Docker can't forward there and falls back to public DNS, where LAN-only names don't exist. The rollout step then dies with lookup k8s.lan.xbazzi.com on 127.0.0.11:53: no such host. TLS verification is unaffected: the apiserver cert carries IP Address:10.67.0.10 in its SANs, so no tls-server-name override and emphatically no insecure-skip-tls-verify is needed. 10.67.0.10 is the etcd-elected control-plane VIP, so it follows whichever node holds it.

The app has no authentication of its own — every route, including the mutating ones and every page showing mentee PII, is open to whatever reaches the Service. Both its Ingresses carry the tinyauth ForwardAuth middleware, and charts/tinyauth/values.yaml lists both hostnames under protectedApps with an explicit usersAllow. tinyauth is allow-by-default for unknown hostnames, so removing either entry silently disables auth rather than failing closed.

devenv scripts

Script	What it does
cluster:up	pulumi:up → talos:sync-config → cluster:apply
cluster:apply	base:apply + db:apply + metrics:apply (three layers in order)
base:apply	helmfile apply — base cluster stack (Cilium, CoreDNS, LB, Traefik+dashboard, storage, portainer-agent)
traefik:restart	Rolling-restart the Traefik deploy — needed after editing an existing dynamic.yaml router's rule in place (fsnotify sometimes misses ConfigMap symlink swaps). Symptom: 404 with no RouterName in the access log after a base:apply that reworded a Host(...) rule.
db:apply	helmfile apply -f helmfile-db.yaml — local-path + CNPG + pg-cluster. PG_MODE=recovery (default) restores from s3://pg-backup/prev/; PG_MODE=standalone runs initdb.
db:down	Delete pg-cluster CR + PVCs; prints S3 ping-pong rotation commands (live/→prev/). See databases-runbook.md.
db:reset	Same as db:down + prints commands to wipe both archive paths. Follow with PG_MODE=standalone db:apply.
metrics:apply	helmfile apply -f helmfile-metrics.yaml — metrics-server + kube-prometheus-stack + Loki + Alloy
apps:deploy	helmfile apply -f helmfile-apps.yaml — user-facing apps
cluster:down	Scale workloads to 0, delete PVCs, then pulumi:down — NAS subdirs under /mnt/ALEXANDRIA/k8s/nfs are preserved (StorageClass onDelete: retain) so a subsequent cluster:up reattaches the data
redeploy	cluster:down then cluster:up
talos:sync-config	Materialize kubeconfig + talosconfig from Pulumi outputs — run this after any pulumi operation that touches talos-secrets, otherwise talosctl throws x509 CA errors
pulumi:up / pulumi:refresh / pulumi:down	pulumi up / refresh / destroy against the prod stack. Reach for pulumi:refresh before pulumi:up when you've done out-of-band qm destroy or edited PVE snippets by hand.
sops:init	Generate .age-key.txt + install helm-secrets/helm-diff plugins
sops:edit <file>	Decrypt-edit-re-encrypt a SOPS file
backup:up / backup:run / backup:ssh / backup:logs	Config-backup VM lifecycle
edge:apply	Configure walle — WireGuard listener + HAProxy :443 passthrough + the road-warrior VPN UDP relay. ansible.cfg's vault_password_file resolves the become password, so no -K needed.
vpn:new-client <name> <ip>	Mint a road-warrior VPN client — keypair + PSK + a ready-to-use wg-quick conf in ./vpn/ (gitignored), and prints what to paste into the OPNsense peer. e.g. vpn:new-client laptop 10.88.1.2. See roadwarrior-vpn.md.
vault:edit / vault:view	Ansible vault for backup role

TLS certs

No cert-manager / ACME issuer runs in-cluster. Traefik's file provider is fed by:

• Cert material (fullchain.pem + private.key) on the TrueNAS NFS share at /mnt/ALEXANDRIA/certs/<domain>/. Mounted read-only into the traefik pod at /certs via a static in-tree NFS PV (declared in k8s/base-infra/traefik/values.yaml).
• Cert mapping (which cert file goes with which domain) as a ConfigMap rendered from that same values file, mounted at /dynamic/tls.yaml.

Directory layout on TrueNAS — one directory per domain, fullchain.pem + private.key inside each:

/mnt/ALEXANDRIA/certs/
├── xbazzi.com/                 # apex + *.xbazzi.com wildcard
│   ├── fullchain.pem
│   └── private.key
├── lan.xbazzi.com/             # *.lan.xbazzi.com wildcard only (no apex SAN)
│   ├── fullchain.pem
│   └── private.key
├── thegrind.dev/               # *.thegrind.dev wildcard
├── dudeget.online/             # *.dudeget.online wildcard
├── blackpeople.ceo/            # apex + *.blackpeople.ceo wildcard
└── gitgud.foo/                 # apex only

Verify SANs on any cert with openssl x509 -in /mnt/ALEXANDRIA/certs/<domain>/fullchain.pem -noout -ext subjectAltName. A wildcard does not cover its own apex — pick a subdomain unless the apex is explicitly listed as a SAN.

The ConfigMap (see extraObjects in base-infra/traefik/values.yaml) carries both the tls.certificates: list (one entry per domain) and an http.routers / http.services block for 16 out-of-cluster backends — LAN infrastructure (garage, opnsense, dns, nas-ui, wap-ui, ipmi-ui, pbs-ui, PVE + Intel-AMT UIs), the swarm-hosted Portainer server, Nextcloud, and one legacy 301 redirect (todo.dudeget.online → vik.thegrind.dev). Migrated from the docker-swarm Traefik file provider — new backends land here. Traefik is started with --providers.file.filename=/dynamic/dynamic.yaml --providers.file.watch=true. File-provider hot-reloads work for added routers; edits to an existing router's rule sometimes get missed by fsnotify (ConfigMap symlink swap) — run traefik:restart to force a reload if a rule edit didn't take effect.

Adding a domain:

1. On TrueNAS: mkdir /mnt/ALEXANDRIA/certs/<domain> and drop fullchain.pem + private.key.
2. Add an entry under tls.certificates: in the ConfigMap and cluster:apply.

Renewals are pure file overwrites on NFS — no cluster:apply needed, because filenames don't change and Traefik re-reads the cert on each TLS handshake.

Ingresses declare a tls: block with the hostname and no secretName — Traefik matches SNI against the loaded certs.

Dataset perms (why renewals don't break perms)

The ALEXANDRIA/certs dataset is exported as an NFS share with mapall: 65532:65532. Every write from any client — you via sudo, OPNsense's ACME hook, a manual cp — lands on disk as 65532:65532 regardless of the client uid. That matches Traefik's runAsUser: 65532, so freshly-renewed files are always readable.

mapall only rewrites uid/gid, not mode. Writers should still chmod 0600 on private.key (0644 on fullchain.pem). See docs/tls-certs-runbook.md for TrueNAS + OPNsense setup.

Storage notes

Storage	Type	VM images	Snippets	Import	Format
local	dir	✗	✓	✓	—
local-lvm	LVM-thin	✓	✗	✗	raw only
iso	dir	ISOs only	✗	✗	—

• VM disks + cloud-init ISOs → local-lvm
• Machine config snippets → local (requires Snippets content type enabled)
• Debian cloud qcow2 for backup-01 → local (requires Import content type, Proxmox VE ≥ 8.4 — used by download.File in pulumi/components/backup_vm.py)

Configure auxiliary hosts (optional)

The backup-01 VM itself is now provisioned by Pulumi (alongside the Talos VMs — see pulumi/components/backup_vm.py and docs/backup-vm-runbook.md). Once pulumi:up has created it, layer Docker + Oxidized + git pipelines on top with:

backup:up

Other aux hosts: extend ansible/inventory/hosts.yml and write a playbook.

Networking

• Talos control-plane VIP: 10.67.0.10, etcd-elected by Talos itself (_per_node_patch's vip field). Point k8s.lan.xbazzi.com at it in DNS.
• LoadBalancer IP pool: 10.67.1.0/27 (Cilium CiliumLoadBalancerIPPool → k8s/charts/cilium-lb).
• A public hostname with a split-horizon rewrite must be bound to BOTH Traefik entrypoints. If AdGuard rewrites foo.xbazzi.com → 10.67.1.0, LAN browsers arrive on the internal websecure entrypoint, so an Ingress annotated only websecure-pub has no router for that Host and returns a bare 404 — while the same URL works fine from off-LAN. Use router.entrypoints: websecure,websecure-pub (as coaching and tinyauth do; charts/websites achieves the same by listing the public host on its internal Ingress too). whoami.xbazzi.com gets away with websecure-pub alone only because it has no rewrite and always hairpins via walle.
• Advertisement: BGP (eBGP, cluster ASN 64513 ↔ OPNsense ASN 64512), all 4 CPs peer with 10.67.0.1:179. OPNsense installs 4-way ECMP FIB entries (net.route.multipath=1). Data plane proven end-to-end with whoami.
• kube-proxy: replaced by Cilium (kubeProxyReplacement: true + cluster.proxy.disabled: true in Talos config).

TODO

Current state

Component	Status	Notes
Talos Kubernetes	✅ Running	4-node all-CP cluster (talos01..04, 10.67.0.11-.14); allowSchedulingOnControlPlanes: true; QGA extension via factory schematic (ce4c98...:v1.13.6)
Talos API VIP	✅ Running	Etcd-elected VIP 10.67.0.10 (Talos owns it, ARP-announced). Point k8s.lan.xbazzi.com here.
Cilium CNI	✅ Running	Helm chart v1.19.6 (bumped from v1.17.4 on 2026-07-22 for hubble-CLI parity); base-infra/cilium/values.yaml; kubeProxyReplacement=true; Talos cluster.proxy.disabled=true. rollOutCiliumPods: true (+ matching flags on operator/hubble) so a ConfigMap change triggers a rolling restart — without it, keys like enable-wireguard/encrypt-node drift per-agent (memory project_cilium_config_drift_per_node).
BGP LoadBalancer	✅ Running	Cilium BGP CRs peer with OPNsense FRR (cluster ASN 64513 ↔ 64512), pool 10.67.1.0/27; OPNsense installs 4-way ECMP FIB (net.route.multipath=1)
Ingress controller	✅ Running	Traefik; helmfile-managed; base-infra/traefik/values.yaml; LB IP 10.67.1.0; TLS via file provider on /mnt/ALEXANDRIA/certs (static NFS PV). HA: 2 replicas, node anti-affinity (required, hostname), PDB minAvailable=1 — each replica's node advertises the LB /32 so OPNsense ECMP fans ingress across both
TLS certs	✅ Running	OPNsense ACME issues; copy_certs_to_nfs.sh (a [copycerts] configd action, push-deployed from the repo via ansible/roles/opnsense_scripts) copies renewals to TrueNAS /mnt/ALEXANDRIA/certs; the Traefik file-provider ConfigMap maps each host cert and watch reloads in-place. See tls-certs-runbook.md.
Persistent storage	✅ Running	csi-driver-nfs against TrueNAS (ALEXANDRIA/k8s/nfs); default StorageClass truenas-nfs; chart-managed
whoami	✅ Running	Local chart charts/whoami; TLS served by Traefik file provider; https://whoami.lan.xbazzi.com
Static websites	✅ Running	Local chart charts/websites; one Caddy pod serving N static sites as vhosts, a git-sync sidecar per repo (push→live ~30s, no redeploy); Traefik does TLS+routing, Caddy is the origin. git-host egress via CCNP allow-websites-git-egress. First sites: xbazzi.com (Sculpin build_production/), blackpeople.ceo. See docs/websites.md.
Vaultwarden	✅ Running	Bitwarden-compatible password manager. Local chart charts/vaultwarden; migrated off TrueNAS 2026-07-27. Backed by per-app CNPG database (cnpg-app-vaultwarden provisions role + DB + creds mirror via chart cnpg-app-db). /data on truenas-nfs PVC (attachments, sends, rsa_key.der). LAN-only at https://vaultwarden.lan.xbazzi.com; public exposure gated on outstanding hardening TODOs. Image pinned to 1.37.0-alpine — 1.34.x and earlier lack /identity/accounts/prelogin/password which modern Bitwarden clients require. Admin argon2 hash SOPS-encrypted at k8s/base-infra/vaultwarden/secrets.yaml.
Coaching	✅ Running	Mentorship scheduling app (FastAPI + Jinja/HTMX + OR-Tools CP-SAT optimizer), source at ~/repos/coaching → gitgud.boo/xbazzi/coaching. Local chart charts/coaching; per-app CNPG database via cnpg-app-coaching. The only CI-deployed release — Forgejo Actions builds the image on push to master, pushes it to the Forgejo package registry on gitgud.boo, and rolls the Deployment out as the RBAC-scoped coaching-deployer ServiceAccount; the migrate initContainer runs alembic upgrade head first. Public at https://coaching.xbazzi.com via walle, also coaching.lan.xbazzi.com — both hostnames must stay under xbazzi.com, since tinyauth's session cookie is scoped to the registrable domain of TINYAUTH_APPURL and a protected app on another root domain redirect-loops forever. The app has zero built-in auth, so both Ingresses sit behind tinyauth with a coach user scoped to these two hosts only. See "CI-driven apps" above.
Portainer agent	✅ Running	Local chart charts/portainer-agent; LoadBalancer on port 9001
Config-backup VM	✅ Running	Now Pulumi-managed (backup_vm.py, protect=True); Oxidized + Ansible pipelines for Brocade / OPNsense / OpenWrt / TrueNAS / Talos (Proxmox temporarily disabled — pmxcfs lock-race)
CoreDNS	✅ Running	Helm-managed (base-infra/coredns/values.yaml); Corefile in git; forwards to OPNsense/AdGuard; .lan stub zone. use_tcp: true on the . zone is required — chart default omits it, which strips TCP-53 from both the Service and container port, so any TCP DNS fallback (Loki S3 lookups, large responses) hits 10.96.0.10:53 with no BPF LB entry and allow-dns can't match it. Anti-affinity selector is k8s-app: coredns (pod label; the chart's isClusterService: true only stamps kube-dns on the Service).
Traefik dashboard	✅ Running	Local chart charts/traefik-dashboard; auth via tinyauth ForwardAuth (cross-ns Middleware ref, same login page as every other admin UI); https://traefik.lan.xbazzi.com
Traefik dynamic routes	✅ Running	ConfigMap dynamic.yaml fronts 16 out-of-cluster backends: garage, opnsense, dns (AdGuard), nas-ui, wap-ui, ipmi-ui, pbs-ui, port (swarm Portainer), per-node PVE UIs (pve{1,2,3}-ui with pve-timeouts + x-forwarded-proto-https), a cluster-wide pve.lan.xbazzi.com round-robining all three PVE nodes with cookie stickiness (VNC/console WebSocket sessions must stay pinned to the vncproxy-owning node), Intel AMT (pve{1,2,3}-atmweb), cloud.dudeget.online (Nextcloud), plus a legacy todo.dudeget.online → vik.thegrind.dev 301 redirect. Migrated from the old docker-swarm Traefik. Vaultwarden dropped from this list — it's now an in-cluster Deployment with its own Ingress.
Local-path storage	✅ Running	Vendored Rancher local-path-provisioner in kube-system; StorageClass local-path; backs CNPG PVCs. Requires kubelet extraMount for /var/local-path-provisioner in _ALL_PATCH
CloudNativePG	✅ Running	Operator + shared 3-instance Postgres 17 cluster (primary + 2 hot-standby); LB IP 10.67.1.5; continuous WAL + daily base backup to Garage pg-backup bucket
metrics-server	✅ Running	kubectl top / HPA; Talos self-signed kubelet certs → --kubelet-insecure-tls
Grafana	✅ Running	State PVC on local-path (dies with cluster); https://grafana.lan.xbazzi.com; adminPassword SOPS-encrypted; datasources: Prometheus (default) + Loki + Tempo (tracesToLogsV2 correlation wired). Auto-provisioned dashboards for Cilium/Hubble/Traefik/Node Exporter come from the individual charts; hand-created dashboards go into k8s/charts/grafana-dashboards/dashboards/ as JSON so they survive rebuilds. Phone-home paths killed via grafana.ini (updates check, reporting, feedback links, news feed, plugin catalog).
Alertmanager	✅ Running	TSDB on local-path; Ingress https://alertmanager.lan.xbazzi.com (no auth — LAN-only); no receivers wired yet — installed but not paging
Hubble UI	✅ Running	Relay + UI enabled in base-infra/cilium/values.yaml; Ingress at https://hubble.lan.xbazzi.com (Traefik file-provider SNI); no port-forward needed. Service map hides host / remote-node / kube-dns / prometheus / cluster.local FQDN flows by default — sparse namespaces look empty even when Relay has flows (use hubble observe -n <ns> instead)
Prometheus	✅ Running	TSDB on local-path; Ingress https://prometheus.lan.xbazzi.com (no auth — LAN-only); initContainer chown -R 1000:2000 /prometheus handles the hostPath+fsGroup gap. Prior subPath-ownership crashloop no longer reproduces on fresh installs — recursive chown covers the subdirectory kubelet creates.
Node-exporter	✅ Running	DaemonSet 4/4; needs hostNetwork/hostPID/hostPath which Talos-default baseline PodSecurity rejects — monitoring namespace is stamped pod-security…/enforce=privileged by a presync hook on the kube-prometheus-stack release (see metrics-stack-runbook.md)
Loki	✅ Running	SingleBinary via grafana/loki chart; chunks on Garage S3 (loki-logs bucket); local-path PVC for WAL + hot index. replication_factor: 1 (chart default 3 breaks a 1-replica ring). Grafana Loki datasource wired.
Alloy	✅ Running	Grafana Alloy DaemonSet; tails /var/log/pods/* via local.file_match + loki.source.file, ships to Loki gateway. Memory limit bumped to 512Mi to absorb transient tailer-buffer spikes on nodes with high pod churn (root cause: k8s discovery producing multiple targets per multi-port pod like CNPG → duplicate tailers; not yet deduped).
Tempo	✅ Running	SingleBinary via grafana/tempo chart; trace blocks on Garage S3 (tempo-traces bucket); local-path PVC for WAL. OTLP gRPC receiver on :4317; Traefik ships every request as a trace (tracing.otlp.grpc.endpoint: tempo.monitoring.svc:4317). Grafana Tempo datasource + tracesToLogsV2 correlation to Loki.
Grafana dashboards chart	✅ Running	Local chart charts/grafana-dashboards; iterates dashboards/*.json and renders one ConfigMap per file with the grafana_dashboard=1 label. Drop any exported dashboard JSON into that directory and metrics:apply to persist it across cluster rebuilds (Grafana state on local-path dies with the cluster).
Network segmentation	✅ Running	23 CCNPs enforcing (k8s/charts/network-policies/) under policyEnforcementMode: always — every endpoint is default-deny structurally, not via a universal-selector allow-list. allow-apiserver is scoped by the apiserver-access=allowed namespace label (stamped by the presync hook in k8s/helmfile.yaml), so app-tier pods without a legit reason can't hit the K8s API. LB pool split into internal 10.67.1.0/28 + public 10.67.1.16/28 via serviceSelector; dual Traefik entrypoints (websecure on internal LB, websecure-pub on public LB); Cilium WireGuard encryption for pod-to-pod + host-to-host traffic across nodes. allow-dns also runs L7 DNS visibility (rules.dns.matchPattern: "*") so hubble_dns_* metrics + the DNS Overview dashboard populate. toFQDNs policies (via the L7 DNS proxy) narrowly allow named external egress like grafana.com for chart dashboard downloads and gitgud.boo/forgejo.lan.xbazzi.com for the websites chart's git-sync sidecars. See docs/network-segmentation.md.
tinyauth	✅ Running	Traefik ForwardAuth login page at https://tinyauth.lan.xbazzi.com; single Middleware in tinyauth ns protects Ingresses across every namespace via tinyauth-tinyauth@kubernetescrd. Guards Prometheus, Alertmanager, Hubble UI, Traefik dashboard (Grafana keeps its own login). Traefik providers.kubernetesCRD.allowCrossNamespace: true is required for the cross-ns Middleware ref to resolve — without it router.middlewares annotation silently no-ops. Users blob SOPS-encrypted in k8s/base-infra/tinyauth/secrets.yaml.
WAN exposure	✅ Running	Via walle public VPS (107.174.250.152) — OPNsense is behind CGNAT, so walle is the WAN edge: HAProxy L4 → WireGuard tunnel → OPNsense NAT → traefik-public LB 10.67.1.16. TLS terminates in-cluster (SNI), auth stays in-cluster (tinyauth), walle is a dumb L4 pipe. Live routes: whoami.xbazzi.com, xbazzi.com (apex), blackpeople.ceo, coaching.xbazzi.com, and tinyauth.xbazzi.com (the login page itself — ForwardAuth 302s an off-LAN browser to TINYAUTH_APPURL, so it has to be WAN-reachable once any protected app is public). Add more via a websecure-pub Ingress annotation + Cloudflare A record → walle IP. See docs/walle-edge.md.
Road-warrior VPN	🚧 Config landed, not yet brought up	WireGuard admin VPN for working on the cluster while travelling. Terminates on OPNsense (instance roadwarrior, wg2, 10.88.1.0/24) — walle only relays 51821/udp into the existing tunnel (firewalld DNAT + masquerade scoped to 10.88.0.2), so the crypto endpoint stays in the LAN and the live :443 path needs no change. Public vs. private filtering is unaffected: the relayed packet is filtered while still encrypted on wgedge, then again as plaintext on wgvpn with source 10.88.1.0/24. walle side is Ansible (roles/edge_gateway/tasks/roadwarrior.yml, edge:apply); OPNsense side is manual GUI. Client configs via vpn:new-client <name> <ip>. Split tunnel, MTU 1340 (double encapsulation). See docs/roadwarrior-vpn.md.
Pulumi	✅ Running	Single prod stack against Garage S3 backend (PULUMI_BACKEND_URL in .env); config in pulumi/Pulumi.prod.yaml. pulumi:up / pulumi:down targets --stack prod.
GitOps	❌ Not installed	Next big rock

Next steps

high prio

• set up postfix or other lightweight k8s mail relay

Observability polish

• Wire Alertmanager to a receiver (email / matrix / Discord webhook) — installed but silent today.
• Storage quotas + Alertmanager alerts on Prometheus / CNPG / Loki PVC usage so we get paged before disks fill.
• Add Proxmox viz in Grafana.
• Add OPNsense map in Grafana.
• Dedupe Alloy's k8s discovery output so multi-port pods (CNPG, etc.) produce one file-tailer target instead of N. Root-cause fix for the memory spikes on nodes with heavy pod churn — today the 512Mi limit is a workaround. Likely a keep+hashmod=0 on __meta_kubernetes_pod_uid in the relabel block.
• Add postfix node or some other mail relay
• Consider hubble_http_* L7 visibility on the Traefik→backend path if we ever want per-backend request rates in Grafana beyond what Traefik metrics + Tempo traces already give us.

Security follow-ups from the segmentation rollout (see docs/network-segmentation.md)

• Scope down Portainer agent's cluster-admin RBAC after initial setup.
• Consider (not urgent) Cilium host firewall (hostFirewall.enabled: true in k8s/base-infra/cilium/values.yaml) + a host-level CiliumClusterwideNetworkPolicy selecting reserved:host, allow-listing: SSH from admin VLAN (10.29.90.0/24), kube-apiserver :6443 from admin + remote-node + host, Talos API :50000 from admin + remote-node, kubelet :10250 from remote-node, BGP :179 from OPNsense (10.67.0.1/32), + intra-cluster health via fromEntities: [remote-node, host, health]. Trigger conditions to actually do this: (a) another device joins the K8S VLAN (jump host, sibling VM cluster sharing L2); (b) untrusted workloads land in-cluster (public CI runners, multi-tenant apps, self-hosted GitHub Actions runners for external PRs); (c) you want cheap paranoia now to avoid retrofitting later. Not urgent today because the K8S VLAN is single-tenant Talos-only, no untrusted DaemonSets, and OPNsense + pod-level CCNPs already cover cross-VLAN + pod-to-pod attacks. What this closes when it becomes relevant: L2-adjacent attackers on the K8S VLAN reaching node ports directly (OPNsense doesn't see intra-VLAN); node-to-node lateral movement after a compromise (bypasses OPNsense + CCNPs); hostNetwork pods (Alloy, Cilium agent, csi-driver-nfs) whose egress isn't governed by pod policies. Rollout note when doing it: apply the CCNP first (zero effect while hostFirewall: false), then flip the flag in a separate apply so rollback is one talosctl patch mc / helm rollback away — misconfig can hard-fail Cilium/BGP and lock out kubectl/talosctl.

Migrate off Docker Swarm — see docs/swarm-migration.md

• Blocker: bump Talos VM disk (disk_gb: 30 per node today in pulumi/Pulumi.prod.yaml; target 100+ GB) — Prometheus TSDB + CNPG PVCs + Loki WAL already overcommit; no room for app data until this lands.
• Then follow the migration order in swarm-migration.md: vaultwarden (done — see charts/vaultwarden) → vikunja → nextcloud (if keeping) → forgejo/gitea → postfix (if keeping) → retire swarm postgres → consolidate Harbor + Portainer onto external VM → decommission swarm.
• Public apps are live NOW (xbazzi.com, blackpeople.ceo, whoami.xbazzi.com via walle). Ratelimit landed; the rest of the hardening list still open:
  • Traefik Headers middleware for public routes: HSTS, X-Content-Type-Options, X-Frame-Options, Referrer-Policy, CSP. Baseline OWASP compliance for any WAN-exposed app.
  • OPNsense IPS (Suricata) or CrowdSec plugin — fail2ban-equivalent on the WAN interface to auto-ban abusive IPs. Now that WAN is live via walle these apply to real traffic.
  • Verify pg-backup and loki-logs Garage buckets are backed by a ZFS-encrypted TrueNAS dataset. Both now contain vaultwarden secrets (CNPG dumps include the vaultwarden DB with hashed master keys) — encryption at rest matters immediately.

Backups

• Re-enable Proxmox config-backup (proxmox_backup_enabled: true in ansible/inventory/group_vars/backup/main.yml) once the pmxcfs /etc/pve/priv/lock* exclusion in ansible/roles/backup_targets/tasks/proxmox_one.yml is verified across several runs.

Big-ticket

• GitOps (FluxCD or ArgoCD) — expose UI via Ingress once chosen; replaces apps:deploy / metrics:apply / db:apply scripts. Next major architectural change.
• Keepalived VIP for Proxmox API HA (proxmox.lan.xbazzi.com). Can adguard do this high availability for us by having the same record point to all three node IPs?