> ## Documentation Index
> Fetch the complete documentation index at: https://mezmo-9a59581a-promptless-aura-security-data-handling.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Tracing & Span Layout

> Enable OpenTelemetry, understand AURA's span layout, and interpret OpenInference span kinds.

## Enabling tracing

OpenTelemetry support is enabled by default via the `otel` feature. Configure
your OTLP endpoint using standard environment variables (for example
`OTEL_EXPORTER_OTLP_ENDPOINT`) to export traces — no endpoint set means no
OTel layer is installed. Before you set an endpoint, review [Content
recording](#content-recording) and [Security and Data Handling](/aura/security),
because exported spans can include prompt, response, and tool content.

AURA emits spans using the [OpenInference](https://github.com/Arize-ai/openinference/tree/main/spec)
semantic convention (`llm.*`, `tool.*`, `input.*`, `output.*`) rather than the
`gen_ai.*` conventions. Any `gen_ai.*` attributes from underlying provider
libraries (Rig.rs) are automatically translated to OpenInference equivalents
at export time. This makes AURA traces natively compatible with
[Phoenix](https://github.com/Arize-ai/phoenix) and other OpenInference-aware
observability tools.

Set `PHOENIX_PROJECT_NAME` to choose which Phoenix project receives your
traces. It sets the resource attribute `openinference.project.name`, which is
the only attribute Phoenix uses to route a trace to a project; Phoenix ignores
`service.name` and `OTEL_SERVICE_NAME` for routing. When you leave
`PHOENIX_PROJECT_NAME` unset, AURA sets `openinference.project.name` to the
service name instead (`OTEL_SERVICE_NAME`, then `aura`), so your traces are
grouped under a project of that name rather than one you chose. In a
multi-replica deployment, set the same `PHOENIX_PROJECT_NAME` value on every
replica or pod in an environment so their traces group into one Phoenix
project; do not template it from a per-pod value such as the pod name or
hostname, or each pod's traces land in a separate project. See the [CLI
reference](/aura/cli-reference) for the full OpenTelemetry environment-variable
table.

## Trace structure

Every request produces two traces:

1. **HTTP trace** — covers the request/response lifecycle
2. **Agent trace** — covers the LLM/tool execution

The agent trace is rooted at `agent.stream` with `parent: None` so Phoenix
sees it as an independent trace root with all LLM I/O attributes.

### HTTP trace (both modes)

```text theme={null}
chat_completions (CHAIN)
  └── streaming_completion (CHAIN)
```

### Single-agent mode

```text theme={null}
agent.stream (LLM, ROOT)
  └── agent.turn (LLM)
      ├── execute_tool (TOOL)
      │   └── mcp.tool_call (TOOL)
      └── execute_tool (TOOL)
          └── mcp.tool_call (TOOL)
```

### Orchestration mode

```text theme={null}
agent.stream (LLM, ROOT)
  └── orchestration (CHAIN)
        ├── orchestration.planning (LLM)
        │   └── agent.turn (LLM) → execute_tool → mcp.tool_call
        └── orchestration.iteration (CHAIN)
            └── orchestration.worker (LLM)
                └── agent.turn (LLM) → execute_tool → mcp.tool_call
```

RAG-enabled agents emit an additional `vector.search (RETRIEVER)` span under
`execute_tool`, alongside `mcp.tool_call`, wherever the agent calls a
vector-search tool.

## Span attributes

*When `OTEL_RECORD_CONTENT=true`, AURA also records the assembled system prompt on the agent entry-point spans (`agent.stream`, `agent.prompt`, `agent.chat`) and on the `orchestration.planning` and `orchestration.worker` spans. See [Content recording](#content-recording).*

### Agent root (`agent.stream`)

`user.id`, `session.id`, `metadata`, `input.value`, `output.value`,
`aura.version`, `aura.mode`.

`aura.version` and `aura.mode` are stamped on the streaming root so you can
filter traces by release and by serving path. `service.version` is set on the
tracer resource.

### Agent and LLM spans

AURA emits the following OpenInference attributes on the agent and LLM spans so
Phoenix renders them:

* `llm.provider` is recorded beside `llm.system`. Phoenix's pricing table
  matches on provider and model together.
* `agent.name` is recorded on the `agent.turn` span, translated from
  `gen_ai.agent.name`, whenever the agent, coordinator, or worker is built with
  a name. It is what stops Phoenix from showing "Unnamed Agent".
* `llm.invocation_parameters` comes from the effective LLM config.
* `llm.tools.{i}.tool.json_schema` holds the MCP tool schemas each agent
  advertises, filtered per worker.
* `user.id` comes from the request `user` field, and `metadata` comes from the
  request metadata map. See the [web server reference](/aura/web-server-reference)
  for those request fields.

### Orchestration spans

| Span                      | Attributes                                                                                                                             |
| ------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- |
| `orchestration`           | `orchestration.goal`, `orchestration.max_iterations`, `orchestration.routing` (direct/clarification/orchestrated)                      |
| `orchestration.planning`  | `orchestration.phase`                                                                                                                  |
| `orchestration.iteration` | `orchestration.iteration`, `orchestration.task_count`, `orchestration.quality_score`, `orchestration.will_replan`                      |
| `orchestration.worker`    | `orchestration.task_id`, `orchestration.worker`, `orchestration.task`, `llm.prompt_template.template`, `llm.prompt_template.variables` |

Per-turn prompt and completion token counts (`llm.token_count.prompt` and
`llm.token_count.completion`) live on each `agent.turn` span, recorded by Rig.
From those turn counts:

* Phoenix rolls the turn counts up as the single cost anchor, so every aggregate
  (per worker, per phase, per trace) is computed from the turns.
* AURA's agent-level spans carry no token counts, so they price at zero and
  leave the `agent.turn` totals intact.
* Per-worker LLM overrides price at their own rate because provider and model
  are stamped on each worker's turn spans, through the existing
  `[orchestration.worker.<name>.llm]` config; no new config is needed.

### Retriever spans

`vector.search` spans export as RETRIEVER kind and carry
`retrieval.documents.{i}.*` attributes: the documents returned by a vector
search. Their content follows the `OTEL_RECORD_CONTENT` gate like other recorded
content.

### Tool spans (`execute_tool`)

For HITL-gated tool calls, `execute_tool` carries a `decision_id` attribute
whose value is the UUID of the approval decision that gated the call. This is
the same `decision_id` that appears on the approval webhook payload, the SSE
lifecycle events, and the approval ingress endpoint (see
[Human-in-the-Loop Approval Gates](/aura/hitl)).

The attribute is stamped before the approval outcome is known, so every gated
call carries it regardless of outcome — `approved`, `denied`, `timed_out`,
`cancelled`, or `errored`. It is stamped for gated calls only; ungated tool
calls carry no `decision_id` attribute.

`decision_id` sits on the `execute_tool` (TOOL) parent, not on the nested
`mcp.tool_call` child, where the tool's own status and errors land.

## OpenInference span kinds

| Kind          | Spans                                                                                                                                  |
| ------------- | -------------------------------------------------------------------------------------------------------------------------------------- |
| **LLM**       | `chat`, `chat_streaming`, `agent.turn`, `agent.stream`, `agent.prompt`, `agent.chat`, `orchestration.planning`, `orchestration.worker` |
| **TOOL**      | `execute_tool`, `mcp.tool_call`                                                                                                        |
| **RETRIEVER** | `vector.search`                                                                                                                        |
| **CHAIN**     | `chat_completions`, `streaming_completion`, `orchestration`, `orchestration.iteration`                                                 |

Each span's kind follows from its span name. The agent-level spans now map to
LLM so Phoenix renders their tools and invocation parameters, though they carry
no token counts (see the cost note above).

`chat`, `agent.prompt`, and `agent.chat` are non-streaming entry points; the trace diagrams above cover the streaming path, which is the one AURA's HTTP/CLI backends use.

<Warning title="Breaking change: token-count attributes removed">
  `llm.token_count.total` and `llm.token_count.tool_completion` are no longer
  emitted on any AURA-owned span. The per-turn counts `llm.token_count.prompt`
  and `llm.token_count.completion` on each `agent.turn` span replace them. If you
  have dashboards or alerts keyed on those two attributes on AURA spans, point
  them at `llm.token_count.prompt` and `llm.token_count.completion` on
  `agent.turn` before upgrading.
</Warning>

## Span parenting

* `agent.stream` is created with `parent: None` to break the link from the
  HTTP handler trace, making it an independent trace root in Phoenix.
* The producer task is instrumented with `agent.stream` so
  Rig's `agent.turn` becomes a direct child.
* In orchestration mode, the orchestrator instruments its spawned task
  with the `agent.stream` span so all orchestration child spans nest under the
  trace root.
* Tool execution propagates the current span into its background task so
  `mcp.tool_call` nests under Rig's `execute_tool`.

## Content recording

`OTEL_RECORD_CONTENT` governs only AURA-owned spans. When `false` (the default),
it omits prompt and completion text, and tool arguments and results. When `true`,
it includes them, along with the assembled system prompt (the preamble, which
includes the orchestration instructions and the skills catalog). AURA truncates
all of this content to `OTEL_CONTENT_MAX_LENGTH` (default 1000 bytes, rounded down
to the nearest UTF-8 boundary). At export, the recorded system prompt appears as
leading `system` entries in the input messages (`llm.input_messages`) in Phoenix
and other OpenInference viewers. AURA translates the raw `gen_ai.system_instructions`
key into those leading `system` entries rather than exporting it directly.

Rig-owned spans (`chat`, `chat_streaming`, `agent.turn`, and `execute_tool`)
record content through `gen_ai.*` attributes. The OpenInference exporter
translates and exports these attributes regardless of `OTEL_RECORD_CONTENT`.
Content is truncated to `OTEL_ATTRIBUTE_VALUE_LENGTH_LIMIT` (default 65,536
bytes, rounded down to the nearest UTF-8 boundary), not `OTEL_CONTENT_MAX_LENGTH`. Because of this,
`OTEL_RECORD_CONTENT=false` is not a complete content-suppression control. See [Security and Data Handling](/aura/security)
before you enable OTLP export.

On worker spans, `llm.prompt_template.template` is static and always recorded.
`llm.prompt_template.variables` carries request content, so it follows the
`OTEL_RECORD_CONTENT` gate and the same truncation as other recorded content.

## Error spans

Every error carries an `exception` span event with an `exception.message`
attribute, which is the format Phoenix's exception view expects.

## Known limitations

Tool errors are only recorded on the `mcp.tool_call` child span, not on Rig's
`execute_tool` parent. This is intentional: `mcp.tool_call` is the canonical
TOOL span for Phoenix.