Record Flux Commits and Releases

The Flux client, horopter-flux, runs beside Flux and watches its objects read-only. It records each commit a GitRepository fetches, ended by the first outcome of each Kustomization that applies it, and each HelmRelease install or upgrade, ended by its outcome. Nothing is configured per release. Each step below but pushing a commit is tested against Horopter v0.14.2, chart horopter-flux 0.6.0, and Flux 2.10.0.

GoalCopy Anchor Link

The commit your cluster runs now, and the client's own install, read back as changes with horo; then every commit Flux applies after it.

PrerequisitesCopy Anchor Link

• A cluster running Flux 2, with a GitRepository applied by a Kustomization.
• Horopter running and reachable from the cluster, and a tenant, from Install Horopter in Kubernetes.
• kubectl with access to the flux-system namespace and Horopter's, and jq.
• The in_cluster and horoctl functions from Create a Tenant, defined in your shell.
• horo, from Install Horopter's Command Line, with egress reachable from your machine.

StepsCopy Anchor Link

Set Your ValuesCopy Anchor Link

Set the chart release, a name for this cluster, and the values you kept from Install Horopter in Kubernetes, in the shell you will use for the rest of the page:

export HOROPTER_FLUX_CHART_VERSION=0.6.0
export CLUSTER=prod-eu
export HOROPTER_NAMESPACE=horopter
export HOROPTER_RELEASE=horopter
export HOROPTER_DATABASE_SECRET=horopter-db-app
export TENANT_ID=<your tenant id>
export HOROPTER_INGRESS_URL=http://horopter-ingress.horopter.svc:8080
export HOROPTER_EGRESS_URL=http://localhost:8081

HOROPTER_INGRESS_URL is ingress as the cluster reaches it, and HOROPTER_EGRESS_URL egress as your machine reaches it.

Issue the KeysCopy Anchor Link

The client posts with a producer key, and horo reads with a reader key from HOROPTER_EGRESS_KEY. Each key is printed once, on a line of its own starting horo_:

PRODUCER_KEY=$(horoctl admin keys issue --tenant "$TENANT_ID" --role producer \
  --source "flux-$CLUSTER" | grep -x 'horo_[0-9a-f_]*')
HOROPTER_EGRESS_KEY=$(horoctl admin keys issue --tenant "$TENANT_ID" --role reader \
  --source horo | grep -x 'horo_[0-9a-f_]*')
export HOROPTER_EGRESS_KEY

Issue one producer key for each cluster. Its source, flux-$CLUSTER, is also its locus, so each cluster records its own changes: two clusters applying one commit record two.

Issue it without --subject-pattern. A Kustomization's changes are recorded with each object it applied as the subject, Kind/name or Kind/namespace/name. A key's pattern is a plain glob whose * stops at /, without the {a,b} that egress's subject filter accepts, so no single pattern covers both. A key narrowed to either has its posts refused with 403, and the client drops them (see When Horopter Cannot Take an Event).

Store the Producer KeyCopy Anchor Link

kubectl create secret generic horopter-flux --namespace flux-system \
  --from-literal=key="$PRODUCER_KEY"

secret/horopter-flux created

To rotate the key, update this Secret and restart the client's pod.

Install the ClientCopy Anchor Link

Write a HelmRepository for Horopter's charts, and a HelmRelease that installs horopter-flux against ingress and the Secret:

cat > horopter-flux.yaml << EOF
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: horopter
  namespace: flux-system
spec:
  type: oci
  url: oci://ghcr.io/horopter-dev/charts
  interval: 1h
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: horopter-flux
  namespace: flux-system
spec:
  interval: 1h
  chart:
    spec:
      chart: horopter-flux
      version: "$HOROPTER_FLUX_CHART_VERSION"
      sourceRef:
        kind: HelmRepository
        name: horopter
  values:
    ingress:
      url: $HOROPTER_INGRESS_URL
      existingSecret: horopter-flux
EOF

Commit horopter-flux.yaml to your fleet repository, where a Kustomization applies it. Here it is applied directly, so the YAML above is the YAML under test:

kubectl apply -f horopter-flux.yaml

helmrepository.source.toolkit.fluxcd.io/horopter created
helmrelease.helm.toolkit.fluxcd.io/horopter-flux created

Wait for the release to be ready:

jq -e '.status.conditions[] | select(.type == "Ready") | .status == "True"' \
  < <(kubectl get helmrelease horopter-flux --namespace flux-system -o json)

true

Read It BackCopy Anchor Link

On start, the client records the current state of every source at the times Flux applied it. So the commit your cluster runs now appears at once, at the time Flux fetched it, and the first rollout the client sees is its own install. Wait until that install's change has ended:

horo changes --locus "flux-$CLUSTER" --subject HelmRelease/flux-system/horopter-flux \
  | jq -e --slurp 'any(.status == "completed")'

true

Then list this cluster's changes of the last 30 days:

horo changes --locus "flux-$CLUSTER" --since 720h | jq -r '"\(.status)  \(.subject)  \(.title)"'

completed  GitRepository/flux-system/podinfo  Merge pull request #... from stefanprodan/release-...
completed  HelmRelease/flux-system/horopter-flux  v...

Each line is a change's status, subject and title. A commit's title is its subject as Flux quoted it, which Flux shortens to 50 characters and .... --format table shows the same changes with their start, duration and ref.

Make a ChangeCopy Anchor Link

Push a commit to the repository a GitRepository fetches, and list the changes again once Flux has applied it:

horo changes --locus "flux-$CLUSTER" --since 1h --format table

The new commit is a change titled by its subject, started when Flux fetched it and ended when each Kustomization that applies it finished.

VerificationCopy Anchor Link

horo changes lists one change for the commit each GitRepository holds, titled by its subject, and one for the client's own HelmRelease, titled by the Horopter release it installed.

What It RecordsCopy Anchor Link

The client keeps no state. Its idempotency keys, prefixed with the kube-system Namespace's uid, make posting an event again harmless, so a restart re-posts what it derives from status.

<action> is install or upgrade, and <reason> is the Event's reason in snake case, such as drift_corrected.

Refs and TitlesCopy Anchor Link

A commit change's ref is the Git revision. Its title is the commit subject, from source-controller's NewArtifact Event; once that Event has expired, it is the branch or tag and short SHA, such as main@b1f2477.

A release change's ref is HelmRelease/<namespace>/<name>/<uid>/v<helm revision>. Its title is the chart version, then the app version once the change ends. A retry changes neither chart nor values, so it posts no attempted, and is titled by its event type until it ends.

When Horopter Cannot Take an EventCopy Anchor Link

An event Horopter cannot take for now (no answer, a 5xx or a 429) is retried with backoff. Any other 4xx is an answer a retry cannot change: the event is logged at ERROR and dropped.

What It Does Not RecordCopy Anchor Link

Commits from Git sources only: an OCIRepository or HelmRepository artifact starts no change. The client's other limits, such as what an outage loses, are listed under The Flux Client in Horopter's known limits.

What's Next?Copy Anchor Link

• Show Changes on Your Grafana Dashboards, with the reader key above.