An LLM observability tool records what your agent did: every model call, tool call, and retrieval, with inputs, outputs, tokens, cost, and timing. When a run goes wrong, the trace shows which step broke. The tools differ in how they bill, whether you can run them yourself, how they accept data, and how much evaluation they build on top of the traces. Details below were checked against each provider's official pages in October 2026.
What is the best LLM observability tool for AI agents?
For most teams, start with Langfuse: it is open source, free to self-host, built on OpenTelemetry, and covers tracing, evals, and prompt management. Use LangSmith if you build on LangChain or LangGraph, Braintrust when evals drive your release process, Arize Phoenix for a local-first tool on OpenInference, Laminar to query traces with SQL, W&B Weave if your team already uses Weights & Biases, and Datadog Agent Observability when the rest of your stack is already in Datadog.
| Tool | Consider it when you need… | What to evaluate |
|---|---|---|
| Langfuse | An open-source platform you can self-host, with tracing, evals, and prompts in one place | How fast billable units grow, since each trace, observation, and score counts |
| LangSmith | Tracing and evals that fit LangChain and LangGraph | 14-day retention on base traces, and the cost of keeping the ones you need longer |
| Braintrust | Evals and experiments as the center of the workflow | Data volume in GB against the Pro plan's $249 starting price |
| Arize Phoenix | A tool you can run on your laptop, with broad auto-instrumentation | Whether the Elastic License 2.0 suits how you plan to deploy it |
| Laminar | SQL over your traces, and replaying a failed run | Maturity of the integrations you depend on, as a younger tool |
| W&B Weave | Agent traces next to your existing W&B work | Ingestion cost per MB, and the move to CoreWeave Agent Lens |
| Datadog | Agent traces in the same place as your APM and logs | LLM span volume, and extra cost for retention past 15 days |
How do LLM observability tools compare on pricing and features?
The main differences are the billing unit (traces, units, spans, or gigabytes), whether you can host it yourself, and whether it accepts standard OpenTelemetry data. All figures come from the linked official pages.
| Tool | Free tier | Paid pricing | Open source and self-hosting | OpenTelemetry | Evals | Agent views |
|---|---|---|---|---|---|---|
| Langfuse | Hobby: 50,000 units a month, 30 days of data access, 2 users (pricing) | Core $29 a month for 100,000 units, then $8 per 100,000; Pro $199 a month | MIT outside its enterprise folders; free to self-host (self-hosting) | Built on it (docs) | LLM-as-a-judge, annotation queues, datasets, experiments | Sessions and agent graphs |
| LangSmith | Developer: 1 seat, 5,000 base traces a month (pricing) | Plus $39 per seat a month with 10,000 base traces; per-trace overage rate not stated on the pages we checked | Not open source; self-hosting on Enterprise only | Accepts OTLP traces (docs) | Datasets, experiments, online evals | Threads with Messages, Turns, and Details views |
| Braintrust | Starter: 1 GB of data, 10,000 scores, 14-day retention (pricing) | Pro $249 a month for 5 GB and 50,000 scores, then $3 per GB | SDK is Apache 2.0; self-hosting on Enterprise only (docs) | Accepts OTLP traces (docs) | Experiments, scorers, online scoring, datasets | Trace pages, Topics, and Patterns (logs) |
| Arize Phoenix | Self-host Phoenix; hosted Arize AX Free has 25,000 spans a month, 15-day retention (pricing) | Arize AX Pro $50 a month for 50,000 spans | Source available under Elastic License 2.0; local, Docker, or Kubernetes (get started) | Built on it, with OpenInference instrumentation | LLM, code, and human evals; datasets and experiments | Traces of model calls, retrieval, and tool use |
| Laminar | 1 GB of data, 7-day retention, 1 seat (pricing) | Starter $30 a month for 3 GB, Pro $150 a month for 10 GB | Apache 2.0; self-hostable (repo) | Accepts OTLP over gRPC and HTTP (docs) | Evals on datasets, locally or in CI | Realtime traces, SQL, replay from cache |
| W&B Weave | 1 GB of ingestion a month, up to 5 seats (pricing) | Pro from $60 a month with 1.5 GB, then $0.10 per MB | SDK is Apache 2.0; self-hosting not stated on the pages we checked | Accepts OTLP spans (docs) | LLM judges and custom scorers | Agents view: sessions, turns, LLM and tool calls |
| Datadog | Up to 40,000 LLM spans a month, 15-day retention (pricing) | Pro $160 a month billed annually for 100,000 LLM spans, then $3.50 per 10,000 | Not open source; hosted only | Accepts GenAI semantic convention spans (docs) | Quality and security evaluations, such as prompt injection | Traces with a span per agent step |
What does an AI agent need from an observability tool?
A chatbot makes one model call per message. An agent makes dozens of calls in a loop, chooses tools, and acts on what they return. Consider a support agent given this request:
A customer says they were charged twice for order 4471. Check the payments, refund the duplicate, and reply to the customer.
The run takes 30 steps, refunds the wrong charge, and the reply still sounds confident. To find out why, you need four things:
- Nested steps: each model call, tool call, and tool result as a span under one trace, so you can see the payments lookup returned two charges and which one the model picked.
- Full inputs and outputs: the exact prompt and tool arguments at that step, not a summary.
- Sessions: the customer's earlier messages grouped with this run, since the mistake may start several turns back.
- Evals on real traces: a check that runs on production traffic and flags refunds that do not match the disputed charge, so the next one is caught without someone reading every trace.
Token and cost per step matter too. A loop that retries a failing tool ten times shows up in the trace long before it shows up on the bill. Instrumenting with OpenTelemetry keeps you from being tied to one tool: most of the tools below accept OTLP data, so you can switch or send to two at once.
Langfuse
The default choice for an open-source platform you can run yourself.
Langfuse's tracing is built on OpenTelemetry, and it also offers native Python and JavaScript SDKs. Traces group into sessions for multi-turn work, and agent graphs draw the path an agent took. On top of traces it adds LLM-as-a-judge evals, annotation queues, datasets, experiments, versioned prompts, and a playground.
The code is MIT licensed outside its enterprise folders, and self-hosting is free with Docker Compose or Kubernetes. On Langfuse Cloud, every trace, observation, and score is a billable unit. Hobby includes 50,000 units a month; Core is $29 a month for 100,000, then $8 per 100,000. An agent run with many steps uses many units, so estimate from a real run.
LangSmith
The natural fit when your agent is built on LangChain or LangGraph.
LangSmith is made by the LangChain team and traces LangChain and LangGraph with an environment variable. It does not require LangChain: it lists integrations for OpenAI, Anthropic, CrewAI, the Vercel AI SDK, and others, and accepts OpenTelemetry traces from any agent. Threads show a conversation as messages, turns, or full detail. Datasets, online evals, dashboards, and alerts are built in.
Pricing is per seat plus traces. The free Developer plan has one seat and 5,000 base traces a month; Plus is $39 per seat with 10,000. Base traces are kept 14 days. Extended traces are kept 180 days for an extra fee, which matters if you want to evaluate against last quarter's runs. Self-hosting is on Enterprise only.
Braintrust
A strong option when evals decide what ships.
Braintrust treats production logs and evals as one dataset. Its logs page shows one row per trace, with SQL filters, online scoring, and Topics, which classifies every trace by task, sentiment, and issue. Loop, its built-in assistant, answers questions about your logs in plain language. You can pull a prompt out of a trace into a playground and run an experiment against a dataset. It accepts OpenTelemetry traces.
Pricing is by data processed and scores. Starter is free with 1 GB and 10,000 scores, kept 14 days. Pro is $249 a month with 5 GB and 50,000 scores, then $3 per GB. Users are unlimited on every plan. Self-hosting is Enterprise only. Braintrust also runs a hosted MCP server that can query logs and experiments with SQL.
Arize Phoenix
Worth evaluating when you want to start on your own machine.
Phoenix runs locally with one command, or in Docker or Kubernetes backed by PostgreSQL. It is built on OpenTelemetry and uses OpenInference instrumentation, with auto-instrumentation for LlamaIndex, LangChain, DSPy, Mastra, the Vercel AI SDK, OpenAI, Bedrock, and Anthropic. It adds LLM, code, and human evals, datasets, experiments, and span replay to rerun a model call with different inputs.
Phoenix is licensed under ELv2, the Elastic License 2.0. That allows self-hosting but is not an OSI open source license, so check it against how you plan to deploy. For a managed service, Arize sells Arize AX: free for 25,000 spans a month with 15-day retention, and Pro at $50 a month for 50,000.
Laminar
A candidate when you want to query traces like a database.
Laminar gives raw SQL access to traces from its UI, CLI, or MCP, plus full-text search and custom dashboards. Its debugger records a run so you can replay it from cache and test a fix with an eval. Signals let you describe a problem in plain language and get a Slack or email alert when it appears. It lists first-class support for the AI SDK, the Claude Agent SDK, LangChain, and Browser Use, and its backend accepts OpenTelemetry traces.
The code is Apache 2.0 and can be self-hosted. Laminar Cloud bills by data: the free plan has 1 GB and 7-day retention for one seat, Starter is $30 a month for 3 GB, and Pro is $150 a month for 10 GB with six months of retention.
W&B Weave
A fit when your team already tracks models in Weights & Biases.
Weave traces functions
you mark with @weave.op, and its Agents view shows
sessions, turns, LLM calls, and tool calls. It also accepts
OpenTelemetry spans, so you can send traces without the Weave SDK.
Evaluations use LLM judges and custom scorers.
Weights & Biases is now part of CoreWeave, and the Weave docs name CoreWeave Agent Lens as its successor for agent tracing; both receive the same data. On current pricing, the free plan includes 1 GB of ingestion a month and up to five seats. Pro starts at $60 a month with 1.5 GB, then $0.10 per MB, which is $100 per additional GB. Check which product you are signing up for before you commit.
Datadog Agent Observability
The practical choice when your services are already monitored in Datadog.
Datadog renamed LLM Observability to Agent Observability. Its Python SDK auto-instruments OpenAI, LangChain, Bedrock, and Anthropic, and it accepts spans that follow the OpenTelemetry GenAI semantic conventions. Agent traces live in the same Datadog account as the rest of your monitoring, so there is no second vendor to run. It includes quality and security evaluations, such as prompt injection checks.
Pricing counts only LLM spans. The free tier covers up to 40,000 a month. Pro is $160 a month billed annually ($240 on demand) for 100,000, then $3.50 per 10,000. Traces are kept 15 days; 30, 60, or 90 days cost $6.50, $8, or $9 per 10,000 LLM spans a month. It is hosted only.
Langfuse vs LangSmith vs Braintrust: which should you choose?
Choose by what you need to control:
- Langfuse to own the data. Self-host it for free, or use the cloud and move later.
- LangSmith for LangChain agents. Tracing works with an environment variable, and evals use the same runs.
- Braintrust for an eval-led process. Logs, scores, and experiments share one place, with unlimited users.
Instrument with OpenTelemetry where you can, so trying a second tool is a configuration change.
How do you test an observability tool on your own agent?
Instrument your agent once, send the same runs to two tools, and try to debug real failures in each. Pick 20 to 30 runs, including a few you already know went wrong.
| Check | What a useful result looks like |
|---|---|
| A long run with many tool calls | Every step present and nested correctly, nothing truncated |
| A run you know failed | You can find the failing step in under five minutes |
| A multi-turn conversation | All turns grouped as one session or thread |
| Cost and tokens | Per-step numbers that match your model provider's bill |
| An eval on production traces | A score that flags the known bad runs and passes the good ones |
| Search | Find every run that called one tool with one argument |
Then estimate a month of your real volume in each tool's billing unit. A trace, a span, a unit, and a gigabyte grow at different rates for the same agent, and long agent runs with large tool outputs can push one model much higher than another. Check retention too: a bug reported three weeks later needs traces that still exist.
How do I let my agent read its own traces?
Connect the tool's MCP server to your agent. Braintrust's hosted MCP server signs in with OAuth and can query logs and experiments with SQL, and Laminar exposes SQL over traces through MCP. An agent can then answer questions like “which runs this week refunded the wrong charge?” without you writing the query.
In Dexto, connections accept any custom MCP server, and Dexto discovers the tools that server offers. Add the MCP URL in Connections settings, or ask Dexto to connect it for you. Your agent code never handles the credential, and you can remove the connection at any time. Write tools act with your account's permissions, so review what the server can change before you connect it.
Where does Dexto fit?
An observability tool tells you what an agent did. Changing what it does next is the job of the harness: the instructions, tools, and workspace around the model. That surrounding system is what Dexto provides.
- Tools: connections include more than 50 built-in integrations and any custom MCP server, so a trace query sits next to Slack, Notion, or GitHub in the same agent.
- A computer: Dexto has a persistent cloud computer where it stores files, runs code, and opens browsers. An agent can export a week of failing traces, group them, and keep the analysis for next week.
- Process: a skill holds the rule a trace taught you, such as “confirm the charge ID before refunding.” An agent configured for the job applies it every time.
For the refund example, that means: find the failing step in the trace, write the fix into a skill, and check the next week's runs for the same pattern. Our investigation case study shows the same loop on a harder problem: agents gathered the evidence, and engineers decided what it proved.
Try it in Dexto
Pick one tool from the table, connect its MCP server, and ask Dexto to explain last week's failed runs. Open Dexto or read the quickstart.