K8S Series Part 02 - GitOps with Flux: Deleting the "Works on My Cluster" Problem

In article 01 I said I almost never use kubectl apply to change things. This article is why. The tool that does apply changes is Flux, and the model it enforces - GitOps - is the single decision that makes this cluster maintainable by one person.

The problem GitOps solves

Picture the "normal" way to run self-hosted apps: you SSH in, run some docker run or kubectl apply commands, tweak a config, restart something, and it works. Six months later you need to rebuild, or you've forgotten why a setting is the way it is, and the knowledge lives only in your shell history and your memory.

This is configuration drift: the running system slowly diverges from anything written down. It has nasty properties:

• You can't reliably reproduce the system.
• "What's actually deployed?" has no authoritative answer.
• Every change is a manual, un-reviewed, un-audited action.
• Disaster recovery is "hope I remember everything."

GitOps changes that. The rule is simple:

Git is the single source of truth. The cluster continuously makes itself match Git.

You never push changes to the cluster. You push changes to a Git repository, and an in-cluster agent notices and applies them. The desired state is always the repo; the cluster is a consequence of the repo.

   IMPERATIVE (drift-prone)            DECLARATIVE / GITOPS (self-correcting)

   you ──kubectl apply──► cluster      you ──git push──► repo
        (and you must                                     │
         remember what                            ┌───────┴────────┐
         you did)                                 │  Flux (in       │
                                                  │  cluster) pulls │
                                                  │  & reconciles   │
                                                  └───────┬────────┘
                                                          ▼
                                                       cluster

What Flux actually is

Flux isn't one thing, it's a set of controllers running inside the cluster, each watching a kind of source and reconciling it. The ones that matter day to day:

source-controller       fetches the Git repo (and Helm repos / OCI artifacts)
kustomize-controller     applies plain manifests / Kustomize overlays from the source
helm-controller          installs & upgrades Helm charts (HelmRelease objects)
notification-controller  events, alerts, and webhook-driven reconciliation

The core idea is a reconciliation loop that never stops:

   every interval (and on git push, via webhook):

   ┌──────────────────────────────────────────────────────────┐
   │  1. PULL    fetch the latest commit from the Git repo    │
   │  2. BUILD   render the manifests (Kustomize / Helm values)│
   │  3. DIFF    compare desired (repo) vs live (cluster)     │
   │  4. APPLY   create/update/patch to close the gap         │
   │  5. PRUNE   (optional) delete objects removed from Git   │
   │  6. REPORT  write status back; emit events on failure    │
   └──────────────────────────────────────────────────────────┘
                              ↻ repeat forever

Because it loops forever, Flux is self-healing. If something deletes a Deployment out
of band, the next reconciliation puts it back. If a node comes back after an outage, its
workloads are already declared and get rescheduled. Drift can't accumulate, because the
loop keeps erasing it.

Bootstrapping: how Flux gets installed

Flux installs itself into the cluster and points itself at the repo with one command:

flux bootstrap github \
  --owner=<you> \
  --repository=k3s-safeqbit-local-hq \
  --branch=main \
  --path=clusters/safeqbit-local-hq \
  --personal

This commits Flux's own manifests into the repo (under flux-system/), installs the
controllers, and creates a deploy key so the cluster can pull. From that moment on, Flux
manages Flux too - even the agent is GitOps-managed. The --path tells Flux which
folder in the repo is the entry point for this cluster.

How my repo is laid out

The layout isn't arbitrary - it encodes dependency order, which is the thing that
makes a from-scratch rebuild work in a single pass.

k3s-safeqbit-local-hq/
├── clusters/safeqbit-local-hq/
│   ├── flux-system/                    # Flux's own bootstrap manifests
│   ├── infrastructure-controllers.yaml # Kustomization → infra/controllers
│   ├── infrastructure-configs.yaml     # Kustomization → infra/configs  (dependsOn controllers)
│   └── apps.yaml                        # Kustomization → apps          (dependsOn configs)
│
├── infrastructure/safeqbit-local-hq/
│   ├── controllers/    # the operators & platform charts (HelmReleases)
│   │                   #   metallb, ingress-nginx, cert-manager, sealed-secrets,
│   │                   #   longhorn, cnpg, velero, nfs provisioners, monitoring
│   └── configs/        # custom resources that DEPEND on those controllers
│                       #   MetalLB IP pools, cert-manager ClusterIssuers,
│                       #   SealedSecrets, VolumeSnapshotClass, Velero schedules
│
└── apps/safeqbit-local-hq/   # one folder per application I run
    ├── authentik/  affine/  immich/  netbox/  vaultwarden/  …
    └── kustomization.yaml     # lists which app folders are active

The critical insight is the controllers → configs → apps ordering. Consider
cert-manager: you can't create a ClusterIssuer (a config) until the cert-manager CRDs
exist (a controller). You can't deploy an app that wants a TLS cert until the issuer
exists. Flux's dependsOn enforces exactly that:

# clusters/safeqbit-local-hq/infrastructure-configs.yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: infra-configs
  namespace: flux-system
spec:
  dependsOn:
    - name: infra-controllers      # ◄── don't apply configs until controllers are Ready
  interval: 1h
  path: ./infrastructure/safeqbit-local-hq/configs
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system

On a clean cluster, Flux brings up controllers, waits until they report healthy, then
applies configs, then deploys apps - all from one git push (or one bootstrap). No
manual "apply this, wait, now apply that."

   bootstrap / push
        │
        ▼
   ┌─────────────┐   dependsOn   ┌──────────┐   dependsOn   ┌──────┐
   │ controllers │ ────────────► │  configs │ ────────────► │ apps │
   │ (operators) │   (healthy)   │  (CRs)   │   (healthy)   │      │
   └─────────────┘               └──────────┘               └──────┘

Two ways Flux applies things: Kustomization vs HelmRelease

Kustomization (Flux's own kind, not to be confused with Kustomize the tool) points at
a folder of YAML and applies it, optionally with overlays. I use this for everything
hand-written - my app manifests, the config CRs.

HelmRelease installs and manages a Helm chart declaratively. Instead of
helm install, I commit an object describing the chart, version, and values:

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: ingress-nginx
  namespace: ingress-nginx
spec:
  interval: 1h
  chart:
    spec:
      chart: ingress-nginx
      version: "4.x.x"          # pinned - reproducible installs
      sourceRef:
        kind: HelmRepository
        name: ingress-nginx
        namespace: flux-system
  values:
    controller:
      kind: DaemonSet
      service:
        externalTrafficPolicy: Local

The helm-controller reconciles this exactly like everything else: change the version or a
value in Git, push, and Flux performs the upgrade. No imperative helm upgrade.

Operating Flux day to day

You mostly leave it alone, but these are the commands I use:

flux get kustomizations -A          # are all my Kustomizations applied & healthy?
flux get helmreleases -A            # same for Helm charts
flux get sources git -A             # is the repo being fetched OK?

# stop waiting for the interval - pull & apply NOW
flux reconcile source git flux-system
flux reconcile kustomization apps --with-source

# pause/resume management of one piece (e.g. during a tricky manual migration)
flux suspend kustomization apps
flux resume  kustomization apps

flux suspend deserves a note: during a database migration I sometimes need Flux to
stop re-asserting state while I do something delicate by hand. Suspend the relevant
Kustomization/HelmRelease, do the surgery, resume. Forgetting to suspend means Flux
"helpfully" reverts your manual step mid-operation - ask me how I know.

When something won't reconcile, the status tells you why:

kubectl describe kustomization apps -n flux-system    # Conditions + last error
flux events                                            # recent Flux activity

A common failure: one bad manifest anywhere in a Kustomization's path makes the whole
Kustomization fail to apply (it's all-or-nothing per build). The error usually names the
offending object - for example, a ScheduledBackup with an invalid cron expression once
blocked my entire apps Kustomization until I fixed that one file.

Why this is worth the upfront cost

GitOps is more setup than docker run. What you get back:

• The repo is the documentation. Want to know what's deployed and how it's
  configured? Read the repo. There's no other source.
• Every change is reviewed and reversible. It's a Git commit - diffable, blame-able,
  revertable. Roll back a bad change with git revert.
• Self-healing. Out-of-band changes get corrected automatically.
• Disaster recovery is a procedure, not a prayer. Fresh K3s + this repo + one secret
  key, run bootstrap, and the cluster reassembles itself. (Full DR walkthrough in
  article 09.)

The deal I made with myself: nothing important may exist only in my head or my shell
history. Flux is what enforces that deal.

With the management model in place, the next few articles dig into the platform pieces
Flux deploys - starting with how a request even reaches the cluster.

Resources

This cluster

• k3s-safeqbit-local-hq repo - see clusters/, infrastructure/, and apps/ for the real layout described here

Documentation

• Flux documentation - concepts, components, and guides
• Flux: Kustomization API - including dependsOn and prune
• Flux: HelmRelease API
• Flux bootstrap (GitHub) - how Flux installs itself
• OpenGitOps principles - what "GitOps" actually means
• Helm documentation - the chart packaging Flux drives
• Kustomize - the templating-free overlay tool

Next lesson → In Getting traffic in, we'll follow a request from your browser all the way to a pod, and answer the question every homelabber hits: how do you get a real load balancer when there's no cloud to hand you one?