Observe Horopter

Horopter reports on its own operation: logs on stderr, Prometheus metrics, and OpenTelemetry traces and metrics. This telemetry describes Horopter's own requests and database calls, never the events producers post. This guide is for Horopter v0.14.2.

GoalCopy Anchor Link

Horopter's logs read, its metrics scraped, its traces sent to a collector, and alerts on the signals that mean something is wrong.

PrerequisitesCopy Anchor Link

• horopter-ingress and horopter-egress running, configured through their environment.
• A Prometheus-compatible scraper, an OpenTelemetry collector, or both.

StepsCopy Anchor Link

Read the LogsCopy Anchor Link

Every binary logs JSON to stderr, one object per line. Every request writes one line at INFO with "msg": "request", except GET /healthz and GET /readyz. The Request Line lists its fields. Keys, request bodies, event attributes and client addresses are never logged.

Follow the request lines of a compose stack's ingress:

docker compose logs --no-log-prefix -f ingress | jq -R 'fromjson? | select(.msg == "request")'

--no-log-prefix matters: without it, compose prepends the service's name to each line, which then no longer parses as JSON. Set HOROPTER_LOG_LEVEL to debug, info, warn or error to choose the least severe level logged.

Scrape the MetricsCopy Anchor Link

Both daemons serve Prometheus metrics at /metrics on a listener of their own, on port 9464 by default. It binds localhost, which is reachable only from inside the same container or pod. To scrape it over the network, bind it to every interface:

export OTEL_EXPORTER_PROMETHEUS_HOST=0.0.0.0

If both daemons run in one pod or on one host, give one of them another port with OTEL_EXPORTER_PROMETHEUS_PORT.

The listener needs no credential, so keep it off any network a stranger can reach, and put nothing sensitive in OTEL_RESOURCE_ATTRIBUTES; see What Horopter Leaves to You.

Send Traces to a CollectorCopy Anchor Link

Tracing is off unless a collector is configured. Point every binary at an OTLP/HTTP collector:

export OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318

That turns on traces, and pushes the same metrics over OTLP. To send only traces, set OTEL_EXPORTER_OTLP_TRACES_ENDPOINT instead. A backend should not both scrape a daemon's /metrics and receive its OTLP metrics, or it stores every series twice.

Tracing covers three areas: http, a span for each request; store, a span for each store operation; and sql, a span for each SQL statement. Silence the per-statement spans on egress, where a dashboard refreshing every few seconds turns them into a constant stream that adds little:

export HOROPTER_TRACE_DISABLE=sql

Keep them on ingress, where an append runs several statements in one transaction and a per-statement span is the only way to see which one is slow. HOROPTER_TRACE_DISABLE accepts any name without checking it, so a misspelled area disables nothing.

OTEL_SDK_DISABLED=true turns all telemetry off, the metrics listener included.

Watch IngressCopy Anchor Link

horopter_event_appends_total counts every event submitted, by outcome. Watch the failed outcome: a sustained rate means producers' events are being refused or failing.

horopter_refold_total counts changes Horopter derived again because an event arrived out of order. Alert on its horopter_refold_altered="true" series, the rate at which late events are changing a recorded outcome, status or end.

Watch Read AdmissionCopy Anchor Link

Read Admission explains the two gates egress puts every read through. Alert on:

• horopter_read_shed_total, by horopter_read_shed_gate: a sustained rate means reads are being refused with 503.
• read shed and read too large log lines at WARN, one for each refused read, naming the route, the tenant and the gate's figures.
• read concurrency limit halved at WARN, each time reads found the connection pool congested.

How close egress runs to each gate is horopter_read_admission_in_flight against horopter_read_admission_limit, and horopter_read_memory_reserved_bytes against horopter_read_memory_capacity_bytes. When reads are refused at the concurrency gate, check the memory pool too: reads waiting for memory hold their place under the concurrency limit.

VerificationCopy Anchor Link

• Once ingress has taken one event, its metrics list horopter_event_appends_total. From inside the daemon's network namespace, or with the port published:

  Copy Text

  curl -fsS localhost:9464/metrics | grep horopter_event_appends_total

• With a collector configured, a request to either daemon appears as a trace with an http span, and log lines written during it carry trace_id and span_id.

NotesCopy Anchor Link

The Request LineCopy Anchor Link

For a replayed idempotency key, subject, type and ref are the original event's.

Every Horopter error is a JSON object. The answers no handler wrote, an unmatched route's 404 or 405 and a panic's 500, carry no error.

Every string value is cut at 1 KiB, and a field that was cut gains <field>_truncated: true. An error body over 2 MiB is not read at all: its line has error_truncated: true and no error. credential is added only once the key resolves, so an unauthenticated caller cannot put a prefix of its choosing on the line.

trace_id and span_id link a line to its trace in a backend such as Tempo. A span is active only when tracing is on, so with tracing at its default off no line carries them. Startup and telemetry-failure lines never do, tracing on or off: they are logged outside any request.

When Telemetry FailsCopy Anchor Link

• A metrics listener that cannot start stops the daemon, with an error naming the variable and its value: a port out of range, a host that does not resolve, or an address in use.
• A collector never stops a daemon. If tracing or OTLP metrics cannot be set up, the binary logs one ERROR line naming the cause and carries on without them. That line is the only sign that export is configured but not running.
• A malformed OTEL_EXPORTER_OTLP_* value is logged at ERROR by OpenTelemetry, which then falls back to its own defaults, so the exporter sends somewhere you did not intend. Treat such a line at startup as a broken configuration.
• A collector that is down is logged at WARN. Spans buffered while it is down are lost; metric totals survive, because every push carries running totals.

No Caller-Chosen LabelsCopy Anchor Link

No metric carries a tenant, a ref, an event id or any other value a caller chooses, so no caller can create series without limit. Per-request detail, tenant included, is on the trace spans.