Observe the system

Outcome

Operators can distinguish process liveness from dependency readiness and correlate persisted product events with stable IDs without exposing credentials. Both backend roles export redacted traces and bounded metrics through one sink-neutral OTLP/HTTP route. A provisionable Grafana dashboard and Prometheus-compatible alert rules cover the signals the product can currently emit honestly.

Prerequisites

Choose approved trace and metric destinations, retention, and incident access before setting an OTLP endpoint. The app never sends prompts, outputs, headers, bodies, provider payloads, exception text, or hidden reasoning through infrastructure telemetry. Exporter headers are secret configuration and are never returned by health endpoints.

Steps

1. Query /health/live to test only the process event loop. Do not use liveness as dependency proof.
2. Query /health/ready before admitting traffic or work. Read its bounded checks and code fields; the response intentionally omits connection targets, credentials, and raw provider errors.
3. Set OTEL_EXPORTER_OTLP_ENDPOINT to one approved OTLP/HTTP base URL. Non-loopback endpoints must use HTTPS. If the collector requires authentication, set percent-encoded comma-separated OTEL_EXPORTER_OTLP_HEADERS values in the role's secret manager.
4. Configure the collector to send metrics to a Prometheus-compatible store and traces to the approved trace store. Load the checked-in Grafana dashboard and alert rules from deploy/observability/.
5. Correlate sanitized traces using stable project, run, task, attempt, execution, and trace IDs. Metric labels remain low-cardinality: bounded routes, outcome classes, transport kinds, the two selected sandbox-provider names, and fixed signal classes only.
6. Export approved provider telemetry because provider dashboards are not canonical retention. Store a reasoning summary only when the provider explicitly supplies one; never hidden chain of thought.

The app role exposes redacted /health/live and /health/ready reports. The worker exposes the same paths on its private health listener: liveness is process-only, while readiness requires PostgreSQL, the Temporal poller, and a fresh reconciliation/outbox cursor. The reconciliation window defaults to 90 seconds and is configurable from 5 through 3,600 seconds. Optional runtime or provider activation status is diagnostic and does not make the worker unavailable by itself.

The app readiness report additionally checks database connectivity, the exact repository migration head, and required artifact/auth configuration. In filesystem mode it also verifies that the artifact root is readable and traversable. When OTLP is configured, readiness remains pending until both trace and metric export have succeeded and becomes unavailable after either exporter fails. An absent OTLP endpoint leaves telemetry disabled without constructing a network exporter.

The dashboard covers request/rejection latency, Temporal workflow/activity failures, outbox delivery, workflow-intent outcomes, approval state and age, sandbox reconciliation, attempt/kernel state, artifact publication and finalization, model/tool settlements, model tokens, usage-cost variance, claim-validation age, sandbox lifecycle latency/resources/drift, per-execution kernel output volume, and active/closed SSE and WebSocket connections. The worker rebuilds current state gauges from PostgreSQL every 30 seconds, so process restart does not reset them.

Local Docker and Daytona pass through the same instrumented SandboxProvider. Ready sandboxes use one fixed five-second, 2 KiB probe that reads cgroup CPU/memory and workspace statvfs only. It performs no filesystem or process walk and returns no paths, identities, environment, or user content. Lumen verifies the ExecResult digest and byte accounting, requires exactly one JSON line, and rejects duplicate keys, non-finite values, extra fields, and impossible resource totals. Provider failures and protocol rejection remain visible as separate fixed outcomes.

Kernel output volume is observed after the durable execution path accepts each event. Inline bytes come from normalized output JSON and externalized bytes from the verified staged-output size. One histogram sample represents one execution, while labels contain only terminal outcome and storage class—never execution identity or output content.

Verify

Confirm app readiness fails on database, migration, artifact-root, or configured exporter failure. Make the worker reconciliation state stale and confirm readiness returns 503 with redacted checks while liveness remains 200. Exercise one command, one outbox delivery, one Temporal operation, one SSE reconnect, and one WebSocket close; confirm the dashboard changes and alerts evaluate against the expected bounded labels. Search the sink for canary credentials and sensitive fixture content; any match blocks activation.

Recover

On leakage, stop the sink, revoke exposed credentials, preserve access-controlled incident evidence, fix redaction at the source, and replay only sanitized diagnostics where policy permits.

For alert-specific triage, follow the checked-in docs/runbooks/observability-alerts.md runbook. Do not weaken authentication, fencing, idempotency, or redaction to clear an alert.

Alert response

LumenApiErrorRateHigh

Check app readiness, split errors by bounded route, and roll back the app artifact when a route regressed. Keep schema at the migrated version and verify the five-minute ratio remains below 5% for ten minutes.

LumenOutboxDeliveryUnhealthy

Check worker and Temporal readiness, then split delivery outcomes by operation. Allow classified transient retries; investigate operator-required or claim-lost outcomes before redriving anything.

LumenOutboxOperatorRequired

Pause the affected operation class, inspect its canonical outbox fence and provider state, and reconcile by deterministic identity. Redrive only through the product command after ruling out an already accepted side effect.

LumenTemporalWorkflowFailure

Pause affected workflow admission, classify the sanitized failure, and retain the compatible worker for open histories. Replay representative history before deploying a workflow-code fix.

LumenTemporalActivityFailures

Inspect retry state, the activity dependency, and its canonical idempotency/fence record. Stop automatic work when the result is ambiguous, non-retryable, or consuming budget without progress.

LumenWorkflowIntentUnapplied

Compare aggregate version, run generation, plan revision, and deterministic Temporal destination. Treat a stale fence as a safe rejection; reconcile unexplained unapplied intent before redrive.

LumenSandboxReconciliationDrift

Stop new assignment to a repeatedly drifting provider class, compare canonical lease/generation with provider labels, and retry as a new immutable attempt from the last canonical checkpoint.

LumenTransportErrors

Split SSE from WebSocket errors. Verify proxy limits, Origin/Host policy, ticket consumption, and SSE cursor replay; never weaken authentication or ticket scope to restore connectivity.

LumenOperationalSnapshotStale

Restore worker database access or its control loop, then wait for snapshot age to fall below 90 seconds before trusting current approval, queue, cost, and claim gauges.

LumenApprovalStuck

Inspect the canonical request and immutable attempt. Resolve or expire it through the authenticated approval command and verify the pending count falls on the next snapshot.

LumenOutboxBacklog

Check worker and Temporal readiness, preserve deterministic destination IDs, and use the fenced repair/redrive path. Verify both backlog depth and oldest age recover.

LumenKernelQueueStuck

Check the Temporal poller, runtime availability, execution generation, and cancellation fence. Retry or cancel through product commands rather than changing canonical rows directly.

LumenArtifactFinalizationFailures

Check storage availability and exact manifest verification. Preserve rejected staging evidence and retry idempotently after repair; never overwrite canonical bytes or weaken digest checks.

LumenExternalOperationFailures

Split model from tool outcomes and inspect sanitized traces plus canonical settlements. Reconcile an ambiguous provider result by its existing identity before retrying, with no implicit paid fallback.

LumenSandboxTelemetryFailures

Split by the fixed provider and provider/protocol reason. Re-run the exact image smoke; do not weaken the fixed timeout, output cap, digest check, or strict schema to restore a graph.

LumenSandboxResourcePressure

Confirm cgroup memory/RSS and workspace disk against the selected class. Reduce admission or choose an approved class within the current budget, preserve canonical artifacts, and verify pressure remains below 90% for a full lookback.

LumenSandboxLifecycleLatencyHigh

Split create/start/stop latency by fixed provider. Reconcile ambiguous creation by deterministic labels, pause affected admission, and never introduce an implicit provider fallback.

LumenKernelOutputVolumeHigh

Compare inline and externalized volume with output count and artifact finalization. Interrupt runaway work through the product command, preserve committed evidence, and move large results to paged artifacts rather than weakening limits.

Next task

Prepare backup and restore.