Observability

Prometheus metrics, structured logs, and OpenTelemetry traces for Orca services.

This is a self-hosting guide. The hosted product at https://www.orcapods.ai needs none of this — sign in and go.

Overview

Every Orca binary ships with three observability pillars out of the box:

Pillar	Technology	Endpoint
Metrics	Prometheus	GET /metrics
Logs	go.uber.org/zap	stdout
Traces	OpenTelemetry	OTLP gRPC/HTTP

No instrumentation code is required. The HTTP metrics endpoint is registered by each service process.

---

Public Status Page

The standalone status service serves the public status page and JSON endpoints under /api/*, including GET /api/status, GET /api/incidents, and GET /api/history. The dashboard sidebar and landing footer show a live status indicator by polling the status endpoint every 60 seconds, then link to the status page.

The dashboard reads its status page URL from the VITE_STATUS_URL environment variable; set it to your deployed status service. The landing site reads its status page URL from landing/src/content/site.ts.

Browser reads are controlled by the status service’s STATUS_ALLOWED_ORIGINS allowlist. Set it to a comma-separated list of first-party dashboard and landing origins that may read /api/* cross-origin. An empty value allows same-origin reads only, and a literal * restores wildcard CORS.

STATUS_ALLOWED_ORIGINS=https://dashboard.example.com,https://www.example.com

The poller opens an incident after a component reports degraded or down for INCIDENT_OPEN_AFTER consecutive polls, resolves it after INCIDENT_RESOLVE_AFTER consecutive operational polls, and worsens an open degraded incident when the component becomes down. Defaults are INCIDENT_OPEN_AFTER=2 and INCIDENT_RESOLVE_AFTER=3.

---

Metrics

The conductor, runner, and standalone VirtualFS server expose Prometheus metrics at /metrics:

curl http://localhost:8080/metrics     # Conductor
curl http://localhost:7070/metrics     # Runner
curl http://localhost:8081/metrics     # VirtualFS in make demo

Shared Runtime Metrics

These collectors are registered by runner processes and by conductor components that execute runtime work locally.

Metric	Type	Description
runtime_runs_total{profile,runtime,status}	Counter	Runs processed by profile, runtime, and terminal status
runtime_run_duration_seconds{profile,runtime}	Histogram	End-to-end run duration
runtime_sessions_total{runtime,event}	Counter	Session lifecycle events by runtime
runtime_sessions_active	Gauge	Currently live sessions
runtime_session_lifetime_seconds	Histogram	Session lifetime from create to shutdown
runtime_sidecar_request_duration_seconds{runtime}	Histogram	Sidecar POST /run round-trip duration, including streaming
runtime_sidecar_request_errors_total{runtime,kind}	Counter	Sidecar communication errors by kind
runtime_tool_calls_total{profile,runtime,tool,status}	Counter	Tool calls observed in run streams
runtime_tool_call_duration_seconds{profile,runtime,tool}	Histogram	Time between a tool_call event and its matching tool_result
runtime_tokens_total{profile,runtime,kind}	Counter	Token usage, where kind is input, output, cache_read, or cache_create

Conductor Metrics

Metric	Type	Description
conductor_runner_requests_total{runner_id,method,status}	Counter	Requests sent from the conductor to remote runners
conductor_runner_request_duration_seconds{runner_id,method}	Histogram	Remote runner request duration
conductor_runner_errors_total{runner_id,method,kind}	Counter	Remote runner communication errors
conductor_routing_decisions_total{runner_hash,reason}	Counter	Pool routing decisions by destination runner hash and reason

Planner Metrics

Metric	Type	Description
planner_plan_total{status}	Counter	Workflow runs that reached a terminal status
planner_plan_node_total{status,profile}	Counter	Workflow-run node terminal transitions by status and profile
planner_plan_repair_total{action}	Counter	Workflow-run repair actions, such as retry_node, replace_node, add_dependency, or abort
planner_plan_active_nodes	Gauge	Workflow-run nodes currently dispatched and in flight
planner_plan_duration_seconds	Histogram	Wall-clock workflow-run duration from running to terminal status
planner_run_aborted_total{reason}	Counter	Planner goroutines aborted before execution. Reasons include no_tenant and store_get; any non-zero value means a workflow run stayed pending instead of reaching execution.

VirtualFS Metrics

These collectors are registered by vfs serve. Dispatcher labels keep mount names and statuses bounded, and HTTP path labels use the matched http.ServeMux pattern instead of raw tenant paths.

Metric	Type	Description
vfs_op_total{op,mount,status}	Counter	Dispatcher operations such as ls, stat, cat, write, delete, find, and grep
vfs_op_duration_seconds{op,mount}	Histogram	Dispatcher operation duration
vfs_cache_total{layer,result}	Counter	Index and byte cache lookups by result: hit, miss, or stale
vfs_driver_errors_total{driver,kind}	Counter	Mount driver errors by backend fingerprint and coarse operation kind
vfs_exec_total{status}	Counter	Workspace.Execute calls by terminal status
vfs_exec_duration_seconds	Histogram	Workspace.Execute duration
vfs_http_requests_total{method,path,status}	Counter	HTTP requests handled by the VirtualFS API
vfs_http_request_duration_seconds{method,path}	Histogram	VirtualFS HTTP request duration

Scrape Configuration

# prometheus.yml
scrape_configs:
  - job_name: orca-conductor
    static_configs:
      - targets: ["conductor:8080"]

  - job_name: orca-runner
    static_configs:
      - targets:
          - runner-0.runners.orca.svc:7070
          - runner-1.runners.orca.svc:7070

  - job_name: orca-vfs
    static_configs:
      - targets: ["vfs:8081"]

---

Structured Logs

Orca uses go.uber.org/zap for machine-parseable logging. Most Go services emit JSON by default; vfs serve defaults to a compact console log format and switches to JSON with LOG_FORMAT=json.

The VirtualFS HTTP API emits an http.request access log for every request with request_id, method, routed path, status, duration, and remote address. Handler error sites also log structured warning or error events with the same request_id, such as auth.unauthorized, exec.executor_failed, cat.dispatch_failed, upload.write_failed, and tree.list_failed, so a failed request can be correlated with the underlying auth, dispatcher, cache, presign, upload, or executor error.

The conductor planner emits planner.run_aborted when a workflow run goroutine exits before execution, with plan_id, reason, and error fields. reason is no_tenant when the planner reached storage without tenant context, and store_get for other workflow-store lookup failures. Replay submission failures are logged as planner.replay_submit_failed.

The conductor artifact path emits artifacts.bucket_unavailable at fatal level during startup when S3_BUCKET is configured but the bucket cannot be listed with the configured endpoint and credentials. If INTERNAL_S3_BUCKET is configured, the conductor also probes that internal run-event bucket; failures emit internal_artifacts.bucket_unavailable at fatal level, while non-fatal initialization errors emit internal_artifacts.init_failed and fall back to the primary artifact bucket. S3-backed run-event persistence emits runs.s3.events_degraded once per run when writing the JSONL event object fails; the log includes run_id, object key, buffered_bytes, final, and error.

Runner Sandbox Boot Log Lines

Runner startup logs the sandbox manager state so operators can tell whether each provider was registered or skipped:

Log message	Level	Fields	Meaning
sandbox.configured	info	provider, default	Emitted once for each registered provider in registration order. The first provider has default=true; later providers have default=false.
sandbox.configured	info	provider="(no default)"	A sandbox manager exists but has no registered providers.
sandbox.provider_skipped	info	provider, reason="env key not set"	Stock environment wiring skipped e2b or daytona because its credential variable was not set.
sandbox.stub	info	none	No sandbox manager was configured, so sandbox tools run against the stub path.
worker_substrate.configured	info	substrates, default_substrate, runner_url, image, images_by_substrate	Worker broker configuration, including every substrate this runner serves and its default.
worker_substrate.domain_allowlist	info	domains	Final public-domain egress allowlist passed to the Daytona sandbox-worker substrate. Non-public hosts are omitted before this line is emitted.
worker_substrate.e2b_gaps	warn	no_auto_archive, no_domain_allowlist	E2B workers have neither Daytona-style automatic archive nor the exact-domain egress allowlist; paused orphan disks require out-of-band cleanup and egress is unrestricted.

Sandbox provider-event handling emits sandbox_event.unknown_provider or sandbox_event.undecodable when a delivery is ignored, and sandbox_event.invalidated with provider, sandbox_id, kind, and matching leases after an accepted event is applied. sandbox_event.fanout_shed is a warning that the 32 concurrent fan-out slots were full; the webhook still received 202, but cached liveness for those sandboxes stays stale until the next provider probe.

Sandbox Worker Lifecycle Log Lines

A workerMode: "sandbox" session runs a paid sandbox for its whole life, so these lines are the cost and correctness trail for it:

Log message	Level	Fields	Meaning
worker.launched	info	session_id, substrate, worker_id, runtime, epoch	A worker was booted. epoch fences this generation: a superseded worker polling the same session is answered with a shutdown command.
worker.died	warn	session_id, worker_id, last_output	The lease was found dead and reaped. last_output is the worker’s final log lines, pulled before Close destroys them, and is usually the only trace of a worker that died before it ever polled.
worker.idle_paused	info	session_id, worker_id, idle_for	The runner’s own idle policy paused a worker. Its billing clock stops here and restarts on the next run’s resume.
worker.idle_pause_failed	warn	session_id, worker_id, error	Pause failed; the worker keeps running and keeps billing, and the next scan retries.
worker.idle_pause_unsupported	debug	session_id, substrate	The substrate has no suspend primitive (the process substrate). The worker is left running and is not retried.
worker_substrate.idle_pause_preempted	warn	worker_idle_pause, worker_idle_timeout, effect	Misconfiguration: the provider auto-stop is at or below the runner’s own threshold, so the provider always fires first and the platform’s idle policy never applies. Raise WORKER_IDLE_TIMEOUT above WORKER_IDLE_PAUSE.
worker.closed / worker.close_failed	info / warn	session_id, worker_id	Lease released at session shutdown. A failure here leaks a paid sandbox.
worker.close_all_unavailable	warn	effect	The shutdown sweep could not run, so worker leases were not swept at exit. Signals a wiring regression rather than a runtime fault.
daytona.auto_archive_not_applied	warn	sandbox_id, requested_minutes, reported_minutes	The provider answered the create with a different auto-archive interval than was asked for, so this sandbox keeps its disk far longer than intended. An unrecognised field does not fail a create, which is why this is checked rather than assumed.
worker_substrate.orphan_disk_unreclaimable	warn	cause, effect	WORKER_IDLE_TIMEOUT=0 with auto-archive configured. Auto-archive only applies to stopped sandboxes, so with nothing stopping them an orphan’s disk is never released.
worker_proxy.disabled	warn	reason, impact	Conductor booted without WORKER_TOKEN_HMAC_KEY, so the worker transport is not mounted and sandbox workers have no path to their runner. Fails closed: mounting it unverified would forward anonymous traffic to the runner with the runner-auth secret attached.
worker_proxy.forward_failed	warn	path, error	The conductor could not reach the owning runner. A worker hanging up mid-poll is filtered out, so this means a genuine transport fault.
worker_proxy.inflight_capped	warn	path, max_inflight	A verified session reached its concurrent worker-transport cap; the request returned 429 without being queued.
dynamic.mcp_token_mint_failed	warn	session_id, tenant_present, error	The runner could not mint the sandbox session’s per-run MCP capability, so the run fails before dispatch. An absent tenant is reported as tenant_present: false; the token itself is never logged.
dynamic.mcp_token_minted	debug	session_id, run_id, ttl	A per-run MCP capability was minted. Only its lifetime is logged; the credential itself must never appear in logs.
worker.events.upload_timeout	warn	session_id, run_id, max_duration	An events upload exceeded the 4-hour wall-clock limit. The run is ended with an error and the request returns 408.
worker.events.upload_too_large	warn	session_id, run_id, cap_bytes	An events upload exceeded the 256-MiB total-body cap. The run is ended with an error and the request returns 413.

A provider auto-stop frees compute but not disk: a stopped cloud sandbox still holds its full disk allocation, and only deletion or archiving releases it. Orphans left by a runner that died without releasing its leases therefore accumulate against the provider’s disk quota and eventually fail every new launch with a quota error, even though nothing is running. WORKER_AUTO_ARCHIVE is what bounds that; see the local-development guide.

VirtualFS Boot Log Lines

vfs serve emits the following structured log events during startup for each backend it attempts to initialize:

Log message	Level	Fields	Meaning
vfs.backend.ready	info	kind, bucket, endpoint, mounts	Backend came up successfully; mounts lists the path prefixes bound to it.
vfs.backend.init_failed	warn	kind, bucket, endpoint, error	Backend could not be initialized (e.g., S3 unreachable, missing credentials). Affected mounts are marked unavailable. The server continues to boot.
vfs.driver.ready	info	ready, unavailable	Summary line at end of startup listing per-backend ready and unavailable mount counts.

Per-mount availability is also visible at runtime via GET /vfs/mounts — check the status and reason fields.

{
  "level": "info",
  "ts": 1745582400.123,
  "caller": "runtime/session.go:42",
  "msg": "session created",
  "session_id": "sess-a1b2c3d4-e5f6g7h8",
  "profile": "researcher",
  "runner": "http://runner-1:7070"
}

Log Fields

Common fields included on structured log entries:

Field	Description
level	debug, info, warn, error
ts	Unix timestamp
caller	Source file and line
msg	Human-readable message
session_id	Session identifier (when applicable)
run_id	Run identifier (when applicable)
request_id	HTTP request identifier for VirtualFS API access and handler logs
profile	Profile name (when applicable)

Log Levels

Component	Default Level
Conductor	info
Runner	info
VirtualFS	info
agent-worker	info (via Node.js console)

Set LOG_LEVEL=debug on any Go service for verbose output.

Querying Logs with jq

# All errors
docker compose logs conductor-1 | grep '"level":"error"' | jq .

# Events for a specific run
docker compose logs runner-1 | jq '. | select(.run_id == "run-f3a9b72c")'

# Tool invocations
docker compose logs runner-1 | jq '. | select(.msg == "tool invoked")'

---

Distributed Tracing

Orca exports OpenTelemetry traces in OTLP format. Configure the exporter endpoint:

# OTLP gRPC (default for most Jaeger/Tempo setups)
OTEL_EXPORTER_OTLP_ENDPOINT=http://jaeger:4317

# Or HTTP
OTEL_EXPORTER_OTLP_ENDPOINT=http://jaeger:4318
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf

Trace Coverage

Traces span the full request lifecycle:

Trace context is propagated via standard W3C traceparent headers. VirtualFS adds spans for inbound HTTP requests, dispatcher operations, and Workspace.Execute calls when OTEL_EXPORTER_OTLP_ENDPOINT is set.

Jaeger Docker Compose Integration

# Add to docker-compose.yml
jaeger:
  image: jaegertracing/all-in-one:latest
  ports:
    - "16686:16686"   # UI
    - "4317:4317"     # OTLP gRPC
  networks: [orca]

# Add to conductor and runner services
environment:
  OTEL_EXPORTER_OTLP_ENDPOINT: "http://jaeger:4317"
  OTEL_SERVICE_NAME: "orca-conductor"  # or "orca-runner"

Then visit http://localhost:16686 to see traces.

---

Health Checks

The conductor, runner, VirtualFS, and billing service expose /healthz:

curl http://localhost:8080/healthz
# 200 OK
# ok

curl http://localhost:7070/healthz
# 200 OK
# ok

curl http://localhost:8081/healthz
# 200 OK

curl http://localhost:8091/healthz
# 200 OK

Use these for liveness probes in Kubernetes and load balancer health checks.

---

Grafana Dashboard

The metrics above compose naturally into a Grafana dashboard against your Prometheus data source (Dashboards → Import, or build from scratch). A useful starting layout includes panels for:

• Active sessions per runner
• Run throughput (runs/min)
• Run duration percentiles (p50, p95, p99)
• Token usage rate
• Tool call breakdown
• Sidecar error rate
• Plan node throughput and repair counts

---

Alerting Examples

# Prometheus alert rules
groups:
  - name: orca
    rules:
      - alert: OrcaRunnerDown
        expr: up{job="orca-runner"} == 0
        for: 1m
        annotations:
          summary: "Orca runner {{ $labels.instance }} is down"

      - alert: OrcaHighErrorRate
        expr: |
          sum(rate(runtime_runs_total{status="error"}[5m]))
          / sum(rate(runtime_runs_total[5m])) > 0.1
        for: 5m
        annotations:
          summary: "Orca run error rate above 10%"

      - alert: OrcaHighTokenUsage
        expr: sum(rate(runtime_tokens_total[1h])) > 100000
        annotations:
          summary: "Orca token usage exceeding 100k tokens/hour"

      - alert: OrcaPlannerRepairSpike
        expr: sum(rate(planner_plan_repair_total[5m])) > 0.2
        for: 10m
        annotations:
          summary: "Orca planner repair rate is elevated"

---

Centralized Logging (OTLP)

Beyond stdout, every service can export structured logs over OTLP to any OpenTelemetry-compatible backend, such as SigNoz, Grafana, Datadog, or Honeycomb. OTLP log export is additive: stdout logging is never affected, so whatever log collection your platform already does keeps working.

Ingest path

Go services — zap ──────────────► stdout (your platform's log collection, unchanged)
                  └─► otelzap ──► OTLP logs ──┐
Go services — OTLP traces ────────────────────┼─► OTel Collector ─► backend of choice
Node agent-worker / e2b-bridge — OTLP logs ───┘      (batching, transforms)

Run an OpenTelemetry Collector between the services and your backend. It gives you one place for batching and attribute transforms, and lets you switch backends without touching service configuration.

Environment variables

Variable	Value	Notes
OTEL_EXPORTER_OTLP_ENDPOINT	Your collector, e.g. http://otel-collector:4317	Shared for traces and logs
OTEL_EXPORTER_OTLP_PROTOCOL	grpc	Default; set explicitly
OTEL_SERVICE_NAME	Per service (see table below)	Must be unique per service
OTEL_LOGS_ENABLED	false by default	Gates the OTLP logs exporter; enable per service once your collector is reachable

Service name values:

Service	OTEL_SERVICE_NAME
conductor	conductor
runner	runner
billing	billing
vfs	vfs
agent-worker	agent-worker
e2b-bridge	e2b-bridge

When OTEL_EXPORTER_OTLP_ENDPOINT is unset or OTEL_LOGS_ENABLED=false, the logs exporter is a no-op and the service continues normally. Stdout is never affected by the flag.

Log-to-trace correlation

When OTEL_LOGS_ENABLED=true, the otelzap bridge attaches the active span’s trace_id and span_id to each log record. Any backend that links logs and traces by these fields (most do) gives you click-through from a log line to its trace without manual correlation queries.

Suggested starting points

• Build your primary triage view as a saved log query filtering severity_text IN (ERROR, FATAL) grouped by service.name.
• Retention of around 15 days for logs and 7 days for traces is a reasonable starting point; tune to your volume and budget.