The handbook
TenantsUsers and rolesApplicationsScopesAgentsContext (graphium)CultivatorsChatsTracingAudit log
Concepts

Concepts: Tracing

Tracing records what happened inside every agent turn — the route the LangGraph took, each model call with its tokens and latency, each tool invocation, and any error — and stores it first-party in the instance's Postgres, per tenant, behind RLS. The console surfaces it on Traces and Sessions (ADR-0013, spec 007-agent-tracing).

Chats show you what the user saw. Traces show you why.

Traces

A Trace is one agent turn: one POST /v1/agents/{name}/run or /stream invocation. It carries:

  • agent_name, agent_version, modelwhat answered, under which resolved config.
  • entryrun or stream.
  • statusok, error (an exception reached the runner), or halted (the recursion backstop tripped, ERR.MULTIAGENT.0004).
  • latency_ms, input_tokens / output_tokens / total_tokens — end-to-end cost of the turn; token sums come from the provider's usage report.
  • input / output — the turn's user message and final assistant reply (truncated debugging copies).
  • session_idthe conversation thread (the chat id when the turn ran in a chat).

Observations

Inside a trace, each timed step is an Observation, one of three types:

  • spanone LangGraph node (router, a Flow, tools, …). The execution tree of spans mirrors the graph's path through the turn.
  • generationone LiteLLM completion: the exact messages the model saw, its reply, model id, finish reason, token usage, latency.
  • toolone tool invocation: arguments in, result (or error) out.

Observations nest (parent_observation_id): generations and tools sit under the node that made them. The trace detail page renders this as the execution tree with a relative-time latency bar per step; selecting a step opens the inspector with its full input/output.

Sessions

A Session groups traces that share one conversation thread — session_id equals the dh_chats.id when the turn ran inside a chat (the common case). Sessions are derived by aggregation, never stored: trace count, error count, first/last activity, duration, and token totals. The session page lists the conversation's turns in order and cross-links each to its trace and, for chats, to the chat replay.

How capture works

Capture is in-process and always on (ACASO_TRACING_ENABLED=1 by default). AgentHub buffers the turn's observations in memory and writes them once, after the turn, as a detached task — tracing adds no latency to runs, and a tracing failure can only ever lose a trace, never fail a run.

Payloads are debugging-grade, not archival: each stored input/output is capped at ACASO_TRACING_MAX_FIELD_BYTES (16 KiB default) with an explicit truncation marker. The LangGraph checkpointer remains the canonical conversation store.

Traces older than ACASO_TRACE_RETENTION_DAYS (30 default) are deleted by the daily sweep:

bash
python -m agenthub.jobs.trace_cleanup

Access

Reading traces requires the traces:read scope (acaso_os:admin includes it). Traces contain conversation payloads — treat the scope like audit-log access. Tenant isolation is enforced at the row level (FORCE RLS), same as every tenant-scoped table.

API: GET /v1/traces, GET /v1/traces/{id}, GET /v1/sessions, GET /v1/sessions/{id} — cursor-paginated, with agent / status / time-range filters.

PreviousConceptsChatsNext ConceptsAudit log