Known Limits

Each limit says what happens and what to do. A release that removes a limit removes its entry.

Reads Assume a WAF in Front of EgressCopy Anchor Link

Read admission protects egress's database connections and memory; it is not protection against a flood of requests. Put a WAF, or an equivalent, in front of egress.

• A read larger than the memory pool takes the whole pool. It is cut to the oldest rows that fit, and while it runs every other read on that replica is shed with 503. One client repeating such a read keeps them shed, at a volume no WAF would notice. Raise HOROPTER_EGRESS_READ_MEMORY_BYTES, or narrow the reads: a shorter window, or a filter.
• Slow clients hold read capacity until their response is written. Run egress behind a buffering reverse proxy.

Read Admission explains the gates.

No Admission Control on WritesCopy Anchor Link

Ingress has no rate limit, per-tenant quota or load shedding, so a burst of appends is not shed gracefully, and one busy producer or tenant slows every other on that ingress. An append the store cannot take is answered with a retryable 503. Rate-limit producers in front of ingress, and have them retry a 503 with backoff.

Rebuilding a Tenant Halts Its IngestCopy Anchor Link

--rebuild-tenant holds the tenant's lock for its whole run, so appends to that tenant fail with a retryable 503 until it finishes. It is not batched and reports no progress. Run it when the tenant's writes are quiet, and prefer re-folding one change when you can name it: Rebuild a Tenant's Projection.

Migrating Trusts the LedgerCopy Anchor Link

horopter-migrate and horoctl migrate apply every migration file the applied_migration table does not list, and the files are not safe to apply twice. A database whose schema is ahead of that table cannot migrate: dropping or emptying the table, or restoring a dump that leaves it out, fails with

apply migration "0002_constrain_status_and_generate_end_effective.sql": ERROR: constraint "change_status_vocabulary" for relation "change" already exists (SQLSTATE 42710)

and nothing is applied. A backup taken before the table existed, at schema version 1, is not affected: it migrates as any version 1 database does.

Record the files the schema already carries, then migrate. schema_version holds the number of the last one; the file names are in internal/store/postgres/migrations. In psql, insert one row for every file numbered up to that version:

SELECT version FROM schema_version;

BEGIN;
CREATE TABLE IF NOT EXISTS applied_migration (
    filename   TEXT PRIMARY KEY,
    applied_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
INSERT INTO applied_migration (filename) VALUES
    ('0001_init.sql'),
    ('0002_constrain_status_and_generate_end_effective.sql'),
    ('0003_split_change_start_into_assertion_and_inference.sql')
ON CONFLICT (filename) DO NOTHING;
COMMIT;

The example stops at version 3. horopter-migrate then applies only the files after the last row.

The Last Terminal Event WinsCopy Anchor Link

A change's status is the outcome of the last event that ended it, by event id. When two reporters end one change, one's success can clear the other's failure. horopter-flux also records each Kustomization failure as a point of its own, so a failure stays visible after a later success. To see every outcome a change had, read its events, GET /v1/events?ref=<locus>/<ref> or horo show.

No Management APICopy Anchor Link

Tenants and credentials are managed with horoctl, against the database, and no other way.

The GitHub ActionCopy Anchor Link

horopter-dev/record-action:

• It records run: commands. A step that uses: another action cannot be wrapped.
• Each re-run attempt is its own change, so a failure and its retry are two refs.
• It runs on Linux x64 and arm64, and macOS arm64; there is no Windows or Intel macOS build.
• With an organisation secret, the credential attests only github/<org>; the repository is named only by the ref.

The Flux ClientCopy Anchor Link

Record Flux Commits and Releases:

• Git sources only. Only a GitRepository's artifact starts a commit change.
• No link from a commit to the releases it caused.
• Kubernetes Events expire, after about an hour by default. Changeset detail and Event-only points from a longer outage are lost; changes are recovered from status.
• An outage loses intermediate states. A rollout that finished while the client was down has an approximate start; of several, only the newest is recorded. A Kustomization that failed and then succeeded at one revision during the outage records only the success.
• A reinstall after an uninstall remediation is not recorded, as Helm reinstalls at v1 under the same HelmRelease. Its Uninstall* point is.
• A retry after an orphaned start, with a client restart in between, can take the orphan's ref, and is then recorded on the orphan's closed change.
• A HelmRelease retry is titled by its event type, and carries no chart version, values digest or generation, until it ends.
• A cold start mid-action can record the previous attempt's chart version as attempted; the end corrects the title and attributes.
• A ref whose next event keeps getting a 5xx or 429 holds its later events in memory until that event is taken; other refs keep posting.
• A Kustomization that health-checks a HelmRelease records a stuck release as a failure of every later commit it applies. Leave the release's health to the release change.

The Docs Describe the Latest Release OnlyCopy Anchor Link

These pages, and the rest of the documentation, describe the latest release. Upgrade notes are in each release's notes.