LangSmith vs Arize Phoenix vs Braintrust (2026): Tracing, Evals, and CI for Production Agents

Quick verdict

There is no universal winner because these products optimise different control points in the AI delivery lifecycle.

Choose LangSmith when LangChain or LangGraph is already central to your application and the immediate problem is understanding graph execution, inspecting prompts, and turning production traces into datasets. Its tight framework integration reduces setup and debugging friction. Watch trace retention, scoring volume, and seat costs as usage grows.

Choose Arize Phoenix when open source deployment, OpenTelemetry alignment, or combined ML and LLM observability matters more than a polished proprietary workflow. Phoenix is the strongest foundation here for teams that want to retain operational control. The trade is that a platform team may need to assemble more of the evaluation, storage, and release process around it.

Choose Braintrust when evaluation is part of software delivery, not a periodic quality exercise. It is the clearest fit for teams that want experiments, datasets, scorers, and CI checks to decide whether a prompt or model change can ship.

For an AU or Singapore regulated business, the decision should start with data flow and deployment boundaries. Phoenix deserves early consideration where self hosting is mandatory. Enterprise deployment options from LangSmith and Braintrust must be validated contractually, including region, subprocessors, support access, retention, deletion, and backup location.

At a glance

Decision area	LangSmith	Arize Phoenix	Braintrust
Primary strength	LangChain and LangGraph tracing and debugging	Open, OpenTelemetry aligned tracing and evaluation primitives	Evaluation, experiments, datasets, and CI gates
Best team fit	Product team building mainly on LangGraph	Platform or ML team operating mixed frameworks	AI team with mature automated delivery
Trace experience	Deep graph aware workflow	Capable and portable, less integrated with one agent framework	Useful, but subordinate to evaluation workflow
Evaluation	Strong online and offline evaluation	Flexible building blocks	Core product strength
Production trace to dataset	Yes	Yes	Particularly central to workflow
Prompt and experiment workflow	Strong	Available, with more assembly required	Strong and evaluation led
OpenTelemetry posture	Supported, but not the main reason to buy	Central through OpenInference and OTel conventions	Framework neutral SDK approach
Open source	No	Yes, Apache 2.0 according to project materials	No
Self hosting	Enterprise option, verify terms	Yes	Enterprise or on premises option, verify terms
Release blocking	Possible with custom CI integration	Build it around evaluations	A defining use case
Main risk	Cost and lock in through deep framework coupling	Operating burden and workflow assembly	Opinionated eval model and commercial dependency

This matrix reflects product positioning and the supplied research as at July 2026. Product boundaries and commercial terms change quickly, so confirm them in a proof of concept and current order form.

Why agent observability is not ordinary APM

Traditional application monitoring answers whether a request completed, how long it took, and which service failed. An agent can return HTTP 200 while selecting the wrong tool, leaking irrelevant context into a prompt, retrying until cost explodes, or producing an answer that is fluent but unsupported. Infrastructure can be healthy while behaviour is wrong.

A production agent therefore needs two linked evidence systems:

Traces explain what happened: model calls, tool calls, retrieved context, graph transitions, latency, tokens, errors, and feedback.
Evaluations assess whether the behaviour was acceptable: correctness, groundedness, policy compliance, tool selection, format, latency, cost, or a domain specific outcome.

The useful loop is trace, curate, evaluate, change, compare, and gate. A trace viewer without datasets and repeatable scorers becomes an incident inspection tool. An eval suite without production traces becomes a laboratory that misses real user inputs. The purchasing question is not which dashboard looks best. It is which platform helps your team close that loop with acceptable ownership and cost.

Architecture and integration model

LangSmith

LangSmith is most compelling as the observability and evaluation layer around a LangChain or LangGraph application. Instrumentation can capture graph steps and nested runs with relatively little translation. Developers can inspect inputs, outputs, timing, token usage, errors, and intermediate state, then associate feedback or evaluation results with runs.

A typical architecture sends application traces to LangSmith, stores curated examples in LangSmith datasets, and runs experiments against candidate prompts, models, or graph versions. CI can invoke evaluation jobs, but the exact release policy remains your responsibility. In a LangGraph team, the value is not merely telemetry. It is the shared execution model between framework and debugger.

That coupling is also the architectural risk. If traces, datasets, annotations, and review processes depend on LangSmith concepts, moving frameworks or observability vendors is more than changing an exporter. Keep stable internal identifiers for users, conversations, builds, prompts, and dataset cases. Export important datasets and scores on a schedule.

Arize Phoenix

Phoenix takes a more open instrumentation path. OpenInference conventions and OpenTelemetry make it suitable for applications that span LlamaIndex, DSPy, Bedrock, custom Python services, and other model or agent frameworks. Teams can run Phoenix themselves and route telemetry within infrastructure they control.

A common design instruments application and model operations with OTel compatible spans, sends them to Phoenix for trace analysis and LLM evaluation, and keeps durable business outcomes in the organisation's data platform. Larger organisations may evaluate Arize AX for managed enterprise scale and broader ML observability. Phoenix and Arize AX should not be assumed to have identical features, operations, or pricing.

Phoenix is best treated as a composable observability component, not proof that the full release process is solved. You still need ownership for evaluator code, golden datasets, annotation policy, CI thresholds, alerting, storage sizing, upgrades, backups, and access control.

Braintrust

Braintrust starts with the hypothesis that quality changes should be measured like software changes. Production logs can become test cases, experiments compare candidate implementations, scorers turn requirements into signals, and CI can reject regressions.

The architecture usually centres on Braintrust datasets, experiments, scorers, and logs. Application code records production activity, engineers promote representative failures into datasets, and pull requests run evaluations against a baseline. A release policy then considers score deltas, confidence, latency, and cost.

This model works well when the organisation accepts that evaluator design is product engineering. It works less well if the team wants only passive tracing or cannot maintain representative datasets. A CI gate built on weak examples or unstable model judges creates false confidence.

Detailed product assessment

LangSmith: best for LangGraph development velocity

LangSmith's advantage is context. A generic trace platform sees spans. LangSmith can present the execution of LangChain and LangGraph applications in terms familiar to the people building them. That shortens the path from a bad answer to the responsible node, tool call, prompt, or retrieved document.

Its evaluation capabilities are substantial. Teams can build datasets from production examples, apply human or automated feedback, compare experiments, and inspect failures. For a small team shipping its first serious LangGraph agent, this integrated workflow can be more valuable than maximum portability.

The weak point is economic and architectural concentration. The supplied research cites third party estimates around a US$39 monthly seat plus trace usage, and an anecdote in which aggressive scoring and retention drove a much larger bill. Treat both as directional, not current pricing or a forecast. Scored traces, long retention, high fan out graphs, and repeated evaluations can multiply billable events. Model the actual span shape of your application before signing an annual commitment.

LangSmith is not restricted to LangGraph, but the further your stack moves from the LangChain ecosystem, the less unique its advantage becomes. If mixed frameworks are a strategic requirement, test exporter quality and metadata consistency rather than relying on a happy path demo.

Choose LangSmith if:

LangGraph is the application backbone.
Developer debugging speed is the immediate constraint.
The team wants tracing, datasets, prompt work, and evaluation in one workflow.
A managed service is acceptable and commercial deployment terms meet policy.

Arize Phoenix: best for openness and platform control

Phoenix is attractive to teams that see telemetry portability as infrastructure strategy. Open source code and OTel aligned instrumentation reduce dependence on one hosted interface. This matters for private AI, mixed model stacks, and environments where trace payloads may contain personal, commercially sensitive, or regulated information.

Its scope extends beyond simple trace viewing. Phoenix supports evaluation and analysis workflows, and the broader Arize product family addresses enterprise ML observability. That combination can suit organisations where classical models and generative systems share governance and incident processes.

The trade is ownership. Self hosting transfers control, but it also transfers upgrades, capacity planning, availability, backup, security patching, and incident response. Open source software is not free operations. A regulated organisation also needs to confirm authentication, role design, auditability, encryption, key management, and retention against its own control framework. Do not infer enterprise controls from the repository licence.

Phoenix's interface may feel less like an integrated agent IDE than LangSmith. That can be acceptable if traces are one component in a platform, with CI and datasets maintained in code. It is less attractive if a small product team expects a turnkey quality operations workspace.

Choose Phoenix if:

Self hosting or telemetry portability is a hard requirement.
The stack spans LLM, agent, and conventional ML systems.
The organisation already operates OpenTelemetry infrastructure.
A platform team can own the surrounding evaluation and release workflow.

Braintrust: best for eval driven delivery

Braintrust is strongest when the unit of progress is an experiment, not a dashboard. It encourages teams to define examples and scorers, compare changes, and bring the result into code review. This is the right mental model for prompts and agent policies that change behaviour without producing compile errors.

Production logs and traces still matter because they supply hard cases and regression examples. Braintrust's differentiation is how directly those examples feed an evaluation workflow. Teams with a mature CI culture can treat quality, latency, and cost as release dimensions.

The hard part is not clicking "run eval". It is creating stable tests. LLM judge scores can drift with prompt and model changes. Human labels can disagree. Aggregate averages can hide catastrophic failures in a small but important segment. Good Braintrust adoption requires versioned datasets, evaluator tests, slice analysis, and explicit override procedures.

The supplied research reports third party free tier and paid plan figures, including one million spans on a free tier and a plan around US$249 monthly. These numbers are directional and may refer to different dates or packaging. Verify current included usage, seats, retention, overages, support, and enterprise deployment directly.

Choose Braintrust if:

Pull requests must demonstrate behavioural quality before release.
The team already treats test datasets as maintained product assets.
Experiment comparison is more important than graph specific debugging.
Commercial hosting and deployment terms satisfy governance requirements.

Important alternative: Langfuse

Langfuse belongs on the shortlist when open source deployment, a broad framework footprint, and a polished tracing workflow are required together. It is particularly relevant for EU focused hosting discussions and teams willing to operate its data services.

It does not change this article's core comparison because the decision axis is still useful: LangSmith for LangGraph depth, Phoenix for an open OTel aligned platform, and Braintrust for eval led CI. Langfuse can sit between those positions. Its self hosted architecture may include PostgreSQL, ClickHouse, Redis, and object storage, depending on version and deployment pattern. That means more control and more operational surface. Verify its current licence, architecture, cloud regions, pricing, and evaluation features before selection.

Pricing and total cost of ownership

Sticker price is a poor predictor of observability cost. Build a monthly model using:

Requests and agent runs.
Spans per run, including retries and parallel branches.
Percentage sampled and retained.
Percentage scored online.
Offline evaluation runs per build.
Dataset size and experiment frequency.
Seats for engineers, reviewers, and auditors.
Storage, egress, support, and self hosted operations.

The research snapshot cites LangSmith at roughly US$39 per seat plus about US$0.50 per thousand base traces, Phoenix OSS without licence cost, and Braintrust plans ranging from free allowances to roughly US$249 monthly. It also cites Langfuse allowances and a similar starting paid figure. All are third party or time sensitive figures. They are directional only and must not be published as current contractual pricing without checking official pages.

For hosted products, ask whether scoring changes retention class, whether nested spans are billed individually, and whether evaluation runs count separately from production traces. For Phoenix, calculate infrastructure plus engineering time, on call load, upgrades, backup testing, and security review. At low volume, managed software often wins. At high volume or under strict control requirements, self hosting can become rational, but only if the organisation already has the operating capability.

Run a two week replay using representative traces. Extrapolate p50 and p95 span counts, not just average requests. Agent loops and incident traffic create the costly tail.

Security, ownership, and regulated deployment

Trace data can be more sensitive than application logs. It may contain prompts, retrieved records, tool arguments, model responses, user feedback, authentication context, and hidden system instructions.

For AU Privacy Act, APRA regulated, Singapore PDPA, MAS regulated, healthcare, legal, and public sector contexts, validate:

Exact processing and backup regions.
Subprocessors and support access locations.
Encryption in transit and at rest, plus customer managed key options.
SSO, role based access, service accounts, and audit logs.
Field redaction before export.
Configurable retention, deletion, legal hold, and tenant isolation.
Incident notification and evidence available for assurance.
Whether prompts or traces are used for product training.

Self hosting Phoenix can simplify residency but does not automatically deliver compliance. Hosted enterprise editions may offer stronger certified controls than a lightly maintained internal deployment. Architecture, contracts, and operating evidence must agree.

Use an internal telemetry adapter to redact secrets and high risk fields before any vendor SDK sees them. Preserve correlation IDs and evaluation metadata without copying complete source records into every span.

Migration guidance

Do not begin migration by replacing dashboards. Begin with a vendor neutral event model.

Inventory span types, attributes, datasets, scorers, annotations, prompts, dashboards, alerts, and release gates.
Define canonical IDs for trace, run, conversation, tenant, user pseudonym, build, prompt version, model, dataset case, and evaluator version.
Export datasets and human labels in an open format.
Add OpenTelemetry or an internal instrumentation boundary where practical.
Dual write a sampled workload to old and new platforms.
Compare trace completeness, ordering, token and cost fields, evaluator output, and query performance.
Recreate release gates in shadow mode before enforcing them.
Backfill only data with a clear operational or audit purpose.
Set a rollback window and document which historical links will stop resolving.

Moving from LangSmith requires special attention to LangGraph specific metadata and dataset semantics. Moving from Phoenix requires deciding which self hosted history is worth importing. Moving from Braintrust requires preserving experiment baselines, scorer versions, and CI output. In every case, human labels are usually more valuable than raw historical spans.

Frequently asked questions

Can one platform handle both tracing and evaluation?

Yes. All three cover both to some degree. The difference is centre of gravity. LangSmith centres framework aware debugging, Phoenix centres open telemetry and analysis, and Braintrust centres experiments and release quality.

Should we use LLM as judge evaluation?

Use it as one signal. Calibrate judges against expert labels, version judge prompts and models, test disagreement, and keep deterministic checks for policy, schema, citations, latency, and cost. Never make a high impact release decision from an uncalibrated aggregate score.

Is OpenTelemetry enough?

OTel provides transport and semantic structure, not the complete quality system. You still need datasets, evaluators, review workflow, release policy, and ownership. It is valuable because it reduces instrumentation lock in.

Can we run two tools?

Yes, for example Phoenix for controlled tracing and Braintrust for CI evaluations. The benefit must exceed duplicate instrumentation, inconsistent identifiers, access administration, and cost. Start with a shared canonical trace model.

What should a first production eval suite contain?

Begin with real high value journeys, known failures, policy boundaries, tool selection, groundedness, response schema, latency, and cost. Segment results by customer, language, workflow, model, and risk class. A small representative suite is better than thousands of synthetic easy cases.

Which tool is best for a five person startup?

If it is a LangGraph product, usually LangSmith. If self hosting is a firm customer requirement, Phoenix. If the founding team already ships every behavioural change through eval CI, Braintrust. Revisit the choice as volume, frameworks, and compliance obligations change.

Conclusion

Buy the workflow your team will actually operate. LangSmith offers the shortest route from a LangGraph failure to a fix. Phoenix offers the strongest ownership and portability story. Braintrust offers the clearest path from evaluation result to release decision.

Whichever platform you select, the durable assets are not dashboards. They are clean instrumentation, representative datasets, versioned evaluators, explicit release policy, and a disciplined feedback loop from production. Build those assets so they can survive a vendor change.