LangSmith Evaluation

LangSmith supports two types of evaluations based on when and where they run:

Offline Evaluation

Test before you ship

Run evaluations on curated datasets during development to compare versions, benchmark performance, and catch regressions.

Online Evaluation

Monitor in production

Evaluate real user interactions in real-time to detect issues and measure quality on live traffic.

Evaluation workflow

Offline evaluation flow

1. Create a dataset

   Create a dataset with examples from manually curated test cases, historical production traces, or synthetic data generation.

2. Define evaluators

   Create evaluators to score performance:

   • Human review
   • Code rules
   • LLM-as-judge
   • Pairwise comparison
3. Run an experiment

   Execute your application on the dataset to create an experiment. Configure repetitions, concurrency, and caching to optimize runs.

4. Analyze results

   Compare experiments for benchmarking, unit tests, regression tests, or backtesting.

Online evaluation flow

1. Deploy your application

   Each interaction creates a run without reference outputs.

2. Configure online evaluators

   Set up evaluators to run automatically on production traces: safety checks, format validation, quality heuristics, and reference-free LLM-as-judge. Apply filters and sampling rates to control costs.

3. Monitor in real-time

   Evaluators run automatically on runs or threads, providing real-time monitoring, anomaly detection, and alerting.

4. Establish a feedback loop

   Add failing production traces to your dataset, create targeted evaluators, validate fixes with offline experiments, and redeploy.

For more on the differences between offline and online evaluation, refer to the Evaluation concepts page.

Get started

Evaluation quickstart

Get started with offline evaluation.

Manage datasets

Create and manage datasets for evaluation through the UI or SDK.

Run offline evaluations

Explore evaluation types, techniques, and frameworks for comprehensive testing.

Analyze results

View and analyze evaluation results, compare experiments, filter data, and export findings.

Run online evaluations

Monitor production quality in real-time from the Observability tab.

Follow tutorials

Learn by following step-by-step tutorials, from simple chatbots to complex agent evaluations.

To set up a LangSmith instance, visit the Platform setup section to choose between cloud, hybrid, or self-hosted. All options include observability, evaluation, prompt engineering, and deployment.

---

Edit this page on GitHub or file an issue.

Connect these docs to Claude, VSCode, and more via MCP for real-time answers.