{props.title}
{props.content}
(with configured headers) MCP-->>Agent: Return tool list Note over Agent,MCP: Later, when agent needs a tool... Agent->>MCP: Call tool
(with configured headers) MCP-->>Agent: Return result ``` The following sections show you how to connect a remote MCP server to Agent Builder: * [General configuration](#general-configuration): Step-by-step instructions for connecting any remote MCP server with authentication headers. Use this if you're familiar with MCP servers and want a quick reference. * [Example: Connecting a custom MCP server](#example-connecting-a-custom-mcp-server): A detailed walkthrough using a GitHub-based MCP server as an example. Use this if you want to see a complete end-to-end example with specific authentication details. ## Add a remote MCP server Configure remote MCP servers in your LangSmith [workspace](/langsmith/administration-overview#workspaces):
- Math operations
- Generate random numbers
- Execute SQL queries
- Read wiki structure and contents
- Ask questions
- List pull requests
- Get pull request details
- Create issues and pull requests
- Comment on issues and pull requests
- Read repository files and list directories
- Read emails (optionally include body, filter with search)
- Send email or reply to an existing message
- Create draft emails
- Mark messages as read
- Get a conversation thread
- Apply or create labels
- List mailbox labels
- List events for a date
- Get event details
- Create new events
- Update events
- Delete events
- Create new docs
- Read docs
- Append text
- Replace text
- Get spreadsheets
- Create spreadsheets
- Read ranges
- Write ranges
- Append rows
- Clear ranges
- List teams and team members
- List issues with filters
- Get issue details
- List labels
- Create, update, or delete issues
- Publish a post with optional image or link
- List alert group settings and get details
- List incidents and get details
- Get outlier incidents, past incidents, and related incidents
- List services and get details
- List teams, members, schedules, and users, and get details
- List oncalls
- Manage team members
- Manage schedules
- Manage event orchestration routers
- List issues
- Get issue details
- Update issues
- Exa web search (optionally fetch page contents)
- Exa LinkedIn profile search
- Tavily web search
- Send a direct message to a user
- Post a message to a channel
- Reply in a thread
- Read channel history
- Read thread messages
- Create and list customers, products, payments, and prices
- Create, list, and finalize invoices
- List, update, and delete subscriptions
- Create and list coupons
- Create list, and update disputes
- Search and fetch Stripe resources
- Search documentation
- Read a tweet by ID
- Read recent posts from a list
- Read webpage text content
- Extract image URLs and metadata
- Notify user (for confirmations/updates)
For detailed information on the API endpoints and data models, refer to the [Agent Server API reference](/langsmith/server-api-ref).
(1 CPU, 2Gi per server) | 1 (default) | 6 | 10 | 3 | 15 | | **Queue workers**
(1 CPU, 2Gi per worker) | 1 (default) | 10 | 1 (default) | 5 | 10 | | **`N_JOBS_PER_WORKER`** | 10 (default) | 50 | 10 | 10 | 50 | | **Redis resources** | 2 Gi (default) | 2 Gi (default) | 2 Gi (default) | 2 Gi (default) | 2 Gi (default) | | **Postgres resources** | 2 CPU
8 Gi (default) | 4 CPU
16 Gi memory | 4 CPU
16 Gi | 4 CPU
16 Gi memory | 8 CPU
32 Gi memory | The following sample configurations enable each of these setups. Load levels are defined as: * Low means approximately 5 requests per second * Medium means approximately 50 requests per second * High means approximately 500 requests per second ### Low reads, low writes The default [LangSmith Deployment](/langsmith/deployments) configuration will handle this load. No custom resource configuration is needed here. ### Low reads, high writes You have a high volume of write requests (500 per second) being processed by your deployment, but relatively few read requests (5 per second). For this, we recommend a configuration like this: ```yaml theme={null} # Example configuration for low reads, high writes (5 read/500 write requests per second) api: replicas: 6 resources: requests: cpu: "1" memory: "2Gi" limits: cpu: "2" memory: "4Gi" queue: replicas: 10 resources: requests: cpu: "1" memory: "2Gi" limits: cpu: "2" memory: "4Gi" config: numberOfJobsPerWorker: 50 redis: resources: requests: memory: "2Gi" limits: memory: "2Gi" postgres: resources: requests: cpu: "4" memory: "16Gi" limits: cpu: "8" memory: "32Gi" ``` ### High reads, low writes You have a high volume of read requests (500 per second) but relatively few write requests (5 per second). For this, we recommend a configuration like this: ```yaml theme={null} # Example configuration for high reads, low writes (500 read/5 write requests per second) api: replicas: 10 resources: requests: cpu: "1" memory: "2Gi" limits: cpu: "2" memory: "4Gi" queue: replicas: 1 # Default, minimal write load resources: requests: cpu: "1" memory: "2Gi" limits: cpu: "2" memory: "4Gi" redis: resources: requests: memory: "2Gi" limits: memory: "2Gi" postgres: resources: requests: cpu: "4" memory: "16Gi" limits: cpu: "8" memory: "32Gi" # Consider read replicas for high read scenarios readReplicas: 2 ``` ### Medium reads, medium writes This is a balanced configuration that should handle moderate read and write loads (50 read/50 write requests per second). For this, we recommend a configuration like this: ```yaml theme={null} # Example configuration for medium reads, medium writes (50 read/50 write requests per second) api: replicas: 3 resources: requests: cpu: "1" memory: "2Gi" limits: cpu: "2" memory: "4Gi" queue: replicas: 5 resources: requests: cpu: "1" memory: "2Gi" limits: cpu: "2" memory: "4Gi" redis: resources: requests: memory: "2Gi" limits: memory: "2Gi" postgres: resources: requests: cpu: "4" memory: "16Gi" limits: cpu: "8" memory: "32Gi" ``` ### High reads, high writes You have high volumes of both read and write requests (500 read/500 write requests per second). For this, we recommend a configuration like this: ```yaml theme={null} # Example configuration for high reads, high writes (500 read/500 write requests per second) api: replicas: 15 resources: requests: cpu: "1" memory: "2Gi" limits: cpu: "2" memory: "4Gi" queue: replicas: 10 resources: requests: cpu: "1" memory: "2Gi" limits: cpu: "2" memory: "4Gi" config: numberOfJobsPerWorker: 50 redis: resources: requests: memory: "2Gi" limits: memory: "2Gi" postgres: resources: requests: cpu: "8" memory: "32Gi" limits: cpu: "16" memory: "64Gi" ``` ### Autoscaling If your deployment experiences bursty traffic, you can enable autoscaling to scale the number of API servers and queue workers to handle the load. Here is a sample configuration for autoscaling for high reads and high writes: ```yaml theme={null} api: autoscaling: enabled: true minReplicas: 15 maxReplicas: 25 queue: autoscaling: enabled: true minReplicas: 10 maxReplicas: 20 ```
The remaining fields: `alert_name`, `project_name` and `project_url` optionally add additional context to the alert message. You can find your `project_url` in the browser's URL bar. Copy the portion up to but not including any query parameters.
(deployed)"] A --> B["Customer A Assistant
━━━━━━━━━━━━━
Model: GPT-4
Tone: Legal
Tools: Custom"] A --> C["Customer B Assistant
━━━━━━━━━━━━━
Model: Claude
Tone: Casual
Tools: Standard"] A --> D["Customer C Assistant
━━━━━━━━━━━━━
Model: GPT-3.5
Tone: Formal
Tools: Limited"] style A fill:#4A90E2,stroke:#2E5C8A,stroke-width:3px,color:#fff style B fill:#B3E0F2,stroke:#4A90E2,stroke-width:2px,color:#000 style C fill:#B3E0F2,stroke:#4A90E2,stroke-width:2px,color:#000 style D fill:#B3E0F2,stroke:#4A90E2,stroke-width:2px,color:#000 ``` * **Environment-specific configurations** * Use different models or settings for development, staging, and production. * Test configuration changes in staging before promoting to production. * Reduce costs in non-production environments with smaller models. * **A/B testing and experimentation** * Compare different prompts, models, or parameter settings. * Roll out configuration changes gradually to a subset of users. * Measure performance differences between configuration variants. * **Specialized task variants** * Create domain-specific versions of a general-purpose agent. * Optimize configurations for different languages, regions, or industries. * Maintain consistent graph logic while varying the execution details. ```mermaid theme={null} graph TD A["Graph: writing-agent
(deployed)"] A --> B["Blog Assistant
━━━━━━━━━━━━━
Model: GPT-4
Tone: Formal
Style: Long-form
Tools: SEO optimization"] A --> C["Tweet Assistant
━━━━━━━━━━━━━
Model: GPT-4-mini
Tone: Casual
Style: 280-char limit
Tools: Hashtag suggestions"] A --> D["Email Assistant
━━━━━━━━━━━━━
Model: GPT-4
Tone: Professional
Style: Medium length
Tools: Templates"] style A fill:#4A90E2,stroke:#2E5C8A,stroke-width:3px,color:#fff style B fill:#B3E0F2,stroke:#4A90E2,stroke-width:2px,color:#000 style C fill:#B3E0F2,stroke:#4A90E2,stroke-width:2px,color:#000 style D fill:#B3E0F2,stroke:#4A90E2,stroke-width:2px,color:#000 ``` ## How assistants work with deployments When you deploy a graph with LangSmith Deployment, [Agent Server](/langsmith/agent-server) automatically creates a **default assistant** tied to that graph's default configuration. You can then create additional assistants for the same graph, each with its own configuration. If your deployment defines multiple graphs in [`langgraph.json`](/langsmith/application-structure#configuration-file), each graph gets its own default assistant: ```json theme={null} { "graphs": { "graph_id_1": "path_to_graph_id_1", // default assistant created for graph_id_1 "graph_id_2": "path_to_graph_id_2" // default assistant created for graph_id_2 } } ``` That is, there can be multiple default assistants—one for each graph defined in your deployment. Assistants have several key features: * **[Managed via API and UI](/langsmith/configuration-cloud)**: Create, list, update, version, and get assistants using the Agent Server/LangGraph SDKs or the [LangSmith UI](https://smith.langchain.com). * **One graph, multiple assistants**: A single deployed graph can support multiple assistants, each with different configurations (e.g., prompts, models, tools). * **[Versioned](#versioning) configurations**: Each assistant maintains its own configuration history through versioning. Editing an assistant creates a new version, and you can promote or roll back to any version. * **[Configuration](#configuration) updates without graph changes**: Update prompts, model selection, and other settings through assistant configurations, enabling rapid iteration without modifying or redeploying your graph code.
━━━━━━━━━
Deployed logic] end subgraph config[Configuration] A1[Assistant 1
GPT-4, Formal] A2[Assistant 2
Claude, Casual] end subgraph state[State] T1[Thread 1
User A] T2[Thread 2
User B] end subgraph runs[Runs] A1T1["Run: A1 + T1"] A1T2["Run: A1 + T2"] A2T1["Run: A2 + T1"] end A1 -.-> T1 A1 -.-> T2 A2 -.-> T1 A1T1 --> G A1T2 --> G A2T1 --> G style G fill:#4A90E2,stroke:#2E5C8A,stroke-width:2px,color:#fff style A1 fill:#B3E0F2,stroke:#4A90E2,stroke-width:2px,color:#000 style A2 fill:#B3E0F2,stroke:#4A90E2,stroke-width:2px,color:#000 style T1 fill:#FFE0B3,stroke:#F59E0B,stroke-width:2px,color:#000 style T2 fill:#FFE0B3,stroke:#F59E0B,stroke-width:2px,color:#000 style A1T1 fill:#B3F2C9,stroke:#10B981,stroke-width:2px,color:#000 style A1T2 fill:#B3F2C9,stroke:#10B981,stroke-width:2px,color:#000 style A2T1 fill:#B3F2C9,stroke:#10B981,stroke-width:2px,color:#000 ``` This diagram shows how a **run** combines an assistant with a thread to execute the graph: * **Graph** (blue): The deployed code containing your agent's logic * **Assistants** (light blue): Configuration options (model, prompts, tools) * **Threads** (orange): State containers for conversation history * **Runs** (green): Executions that pair an assistant + thread **Example combinations:** * **Run: A1 + T1**: Assistant 1 configuration applied to User A's conversation * **Run: A1 + T2**: Same assistant serving User B (different conversation) * **Run: A2 + T1**: Different assistant applied to User A's conversation (configuration switch) When executing a run: * Each run may have its own input, configuration overrides, and metadata. * Runs can be stateless (no thread) or stateful (executed on a [thread](/oss/python/langgraph/persistence#threads) for conversation persistence). * Multiple runs can use the same assistant configuration. * The assistant's configuration affects how the underlying graph executes. The Agent Server API provides several endpoints for creating and managing runs. For more details, refer to the [API reference](/langsmith/server-api-ref)). ***
- A single period (`"."`), which will look for local Python packages.
- The directory path where `pyproject.toml`, `setup.py` or `requirements.txt` is located.
For example, if `requirements.txt` is located in the root of the project directory, specify `"./"`. If it's located in a subdirectory called `local_package`, specify `"./local_package"`. Do not specify the string `"requirements.txt"` itself. - A Python package name.
- `./your_package/your_file.py:variable`, where `variable` is an instance of `langgraph.graph.state.CompiledStateGraph`
- `./your_package/your_file.py:make_graph`, where `make_graph` is a function that takes a config dictionary (`langchain_core.runnables.RunnableConfig`) and returns an instance of `langgraph.graph.state.StateGraph` or `langgraph.graph.state.CompiledStateGraph`. See [how to rebuild a graph at runtime](/langsmith/graph-rebuild) for more details.
- `index` (optional): Configuration for semantic search indexing with fields `embed`, `dims`, and optional `fields`.
- `ttl` (optional): Configuration for item expiration. An object with optional fields: `refresh_on_read` (boolean, defaults to `true`), `default_ttl` (float, lifespan in **minutes**; applied to newly created items only; existing items are unchanged; defaults to no expiration), and `sweep_interval_minutes` (integer, how often to check for expired items, defaults to no sweeping).
true: Keep all three tools (skip uninstall).false/ omitted : Uninstall all three tools (default behaviour).list\[str]: Names of tools to retain. Each value must be one of "pip", "setuptools", "wheel".
- `ttl` (optional): Object with `strategy`, `sweep_interval_minutes`, `default_ttl` controlling checkpoint expiry.
- `serde` (optional, 0.5+): Object with `allowed_json_modules` and `pickle_fallback` to tune deserialization behavior.
- `app`: Path to custom Starlette/FastAPI app (e.g., `"./src/agent/webapp.py:app"`). See [custom routes guide](/langsmith/custom-routes).
- `cors`: CORS configuration with fields such as `allow_origins`, `allow_methods`, `allow_headers`, `allow_credentials`, `allow_origin_regex`, `expose_headers`, and `max_age`.
- `configurable_headers`: Define which request headers to expose as configurable values via `includes` / `excludes` patterns.
- `logging_headers`: Mirror of `configurable_headers` for excluding sensitive headers from logs.
- `middleware_order`: Choose how custom middleware and auth interact. `auth_first` runs authentication hooks before custom middleware, while `middleware_first` (default) runs your middleware first.
- `enable_custom_route_auth`: Apply auth checks to routes added through `app`.
- `disable_assistants`, `disable_mcp`, `disable_a2a`, `disable_meta`, `disable_runs`, `disable_store`, `disable_threads`, `disable_ui`, `disable_webhooks`: Disable built-in routes or hooks.
- `mount_prefix`: Prefix for mounted routes (e.g., "/my-deployment/api").
- `env_prefix`: Required prefix for environment variables referenced in header templates (defaults to `LG_WEBHOOK_`).
- `headers`: Static headers to include with webhook requests. Values may contain templates like `${{ env.VAR }}`.
- `url`: URL validation policy with `allowed_domains`, `allowed_ports`, `require_https`, `disable_loopback`, and `max_url_length`.
- `./src/graph.ts:variable`, where `variable` is an instance of [`CompiledStateGraph`](https://reference.langchain.com/python/langgraph/graphs/#langgraph.graph.state.CompiledStateGraph)
- `./src/graph.ts:makeGraph`, where `makeGraph` is a function that takes a config dictionary (`LangGraphRunnableConfig`) and returns an instance of [`StateGraph`](https://reference.langchain.com/python/langgraph/graphs/#langgraph.graph.state.StateGraph) or [`CompiledStateGraph`](https://reference.langchain.com/python/langgraph/graphs/#langgraph.graph.state.CompiledStateGraph). See [how to rebuild a graph at runtime](/langsmith/graph-rebuild) for more details.
- `index` (optional): Configuration for semantic search indexing with fields `embed`, `dims`, and optional `fields`.
- `ttl` (optional): Configuration for item expiration. An object with optional fields: `refresh_on_read` (boolean, defaults to `true`), `default_ttl` (float, lifespan in **minutes**; applied to newly created items only; existing items are unchanged; defaults to no expiration), and `sweep_interval_minutes` (integer, how often to check for expired items, defaults to no sweeping).
- `ttl` (optional): Object with `strategy`, `sweep_interval_minutes`, `default_ttl` controlling checkpoint expiry.
- `serde` (optional, 0.5+): Object with `allowed_json_modules` and `pickle_fallback` to tune deserialization behavior.
- `cors` with `allow_origins`, `allow_methods`, `allow_headers`, `allow_credentials`, `allow_origin_regex`, `expose_headers`, `max_age`.
- `configurable_headers` and `logging_headers` pattern lists.
- `middleware_order` (`auth_first` or `middleware_first`).
- `enable_custom_route_auth` plus the same boolean route toggles as above.
- `env_prefix`: Required prefix for environment variables referenced in header templates (defaults to `LG_WEBHOOK_`).
- `headers`: Static headers to include with webhook requests. Values may contain templates like `${{ env.VAR }}`.
- `url`: URL validation policy with `allowed_domains`, `allowed_ports`, `require_https`, `disable_loopback`, and `max_url_length`.
- Calling LLMs with [LangChain](/oss/python/langchain/overview)
- Tracing LLM calls to OpenAI, Anthropic or models that follow an OpenAI-compliant format with `@traceable`
- Using LangSmith wrappers for [OpenAI](/langsmith/trace-openai) or [Anthropic](/langsmith/trace-anthropic)
- For other model providers, read the [token and cost information guide](/langsmith/log-llm-trace#provide-token-and-cost-information)
PostMessage Messages
``` ## Where custom rendering appears When enabled, your custom rendering will replace the default output view in: * **Experiment comparison view**: When comparing outputs across multiple experiments:Trace to LangSmith SaaS. | Optional
Disable tracing or trace to LangSmith SaaS. | Optional
Disable tracing, trace to LangSmith SaaS, or trace to Self-Hosted LangSmith. | ### Telemetry Agent Server is automatically configured to report telemetry metadata for billing purposes. See the table below for details with respect to each deployment option. | Cloud | Hybrid | Self-Hosted | | --------------------------------- | --------------------------------- | ------------------------------------------------------------------------------------------------------------------------ | | Telemetry sent to LangSmith SaaS. | Telemetry sent to LangSmith SaaS. | Self-reported usage (audit) for air-gapped license key.
Telemetry sent to LangSmith SaaS for LangSmith License Key. | ### Licensing Agent Server is automatically configured to perform license key validation. See the table below for details with respect to each deployment option. | Cloud | Hybrid | Self-Hosted | | --------------------------------------------------- | --------------------------------------------------- | -------------------------------------------------------------------------------- | | LangSmith API Key validated against LangSmith SaaS. | LangSmith API Key validated against LangSmith SaaS. | Air-gapped license key or Platform License Key validated against LangSmith SaaS. | ***
Enabling LangSmith Deployment requires an [Enterprise](https://langchain.com/pricing) plan.
• Free self-hosted
• Enterprise (paid self-hosted) | | Scalability | Self-managed | Auto-scaling of task queues and servers | | Fault-tolerance | Self-managed | Automated retries | | Concurrency Control | Simple threading | Supports double-texting | | Scheduling | None | Cron scheduling | | Monitoring | None | Integrated with LangSmith for observability | | IDE integration | Studio | Studio | ### Is LangSmith open source? No. LangSmith is proprietary software. There is a free, self-hosted version of LangSmith with access to basic features. The Cloud deployment option and the Self-Hosted deployment options are paid services. [Contact our sales team](https://www.langchain.com/contact-sales) to learn more. For more information, see our [LangSmith pricing page](https://www.langchain.com/pricing). ### Does LangGraph work with LLMs that don't support tool calling? Yes! You can use LangGraph with any LLMs. The main reason we use LLMs that support tool calling is that this is often the most convenient way to have the LLM make its decision about what to do. If your LLM does not support tool calling, you can still use it - you just need to write a bit of logic to convert the raw LLM string response to a decision about what to do. ### Does LangGraph work with OSS LLMs? Yes! LangGraph is totally ambivalent to what LLMs are used under the hood. The main reason we use closed LLMs in most of the tutorials is that they seamlessly support tool calling, while OSS LLMs often don't. But tool calling is not necessary (see [this section](#does-langgraph-work-with-llms-that-dont-support-tool-calling)) so you can totally use LangGraph with OSS LLMs. ### Can I use Studio without logging in to LangSmith? Yes! You can use the [development version of Agent Server](/langsmith/local-server) to run the backend locally. This will connect to the Studio frontend hosted as part of LangSmith. If you set an environment variable of `LANGSMITH_TRACING=false`, then no traces will be sent to LangSmith. ### What is a Deployment Run? An Deployment Run is one end-to-end invocation of a LangGraph agent deployed via LangSmith Deployment. Nodes and subgraphs are not charged separately. Calls to other LangGraph agents (through RemoteGraph or the LangGraph SDK or the API directly) are charged separately, to the deployment that hosts the agent being called. An interrupt for human-in-the-loop creates a separate Deployment Run when resuming. ***
Weather for {props.city}
;
};
export default {
weather: WeatherComponent,
};
```
Next, define your UI components in your `langgraph.json` configuration:
```json theme={null}
{
"node_version": "20",
"graphs": {
"agent": "./src/agent/index.ts:graph"
},
"ui": {
"agent": "./src/agent/ui.tsx"
}
}
```
The `ui` section points to the UI components that will be used by graphs. By default, we recommend using the same key as the graph name, but you can split out the components however you like, see [Customise the namespace of UI components](#customise-the-namespace-of-ui-components) for more details.
LangSmith will automatically bundle your UI components code and styles and serve them as external assets that can be loaded by the `LoadExternalComponent` component. Some dependencies such as `react` and `react-dom` will be automatically excluded from the bundle.
CSS and Tailwind 4.x is also supported out of the box, so you can freely use Tailwind classes as well as `shadcn/ui` in your UI components.
Weather for {props.city}
;
};
export default {
weather: WeatherComponent,
};
```
{thread.messages.map((message) => (
);
}
```
Behind the scenes, `LoadExternalComponent` will fetch the JS and CSS for the UI components from LangSmith and render them in a shadow DOM, thus ensuring style isolation from the rest of your application.
## How-to guides
### Provide custom components on the client side
If you already have the components loaded in your client application, you can provide a map of such components to be rendered directly without fetching the UI code from LangSmith.
```tsx theme={null}
const clientComponents = {
weather: WeatherComponent,
};
{message.content}
{values.ui
?.filter((ui) => ui.metadata?.message_id === message.id)
.map((ui) => (
))}
Weather for {props.city}
);
};
```
### Pass additional context to the client components
You can pass additional context to the client components by providing a `meta` prop to the `LoadExternalComponent` component.
```tsx theme={null}
Weather for {props.city} (user: {meta?.userId})
);
};
```
### Streaming UI messages from the server
You can stream UI messages before the node execution is finished by using the `onCustomEvent` callback of the `useStream()` hook. This is especially useful when updating the UI component as the LLM is generating the response.
```tsx theme={null}
import { uiMessageReducer } from "@langchain/langgraph-sdk/react-ui";
const { thread, submit } = useStream({
apiUrl: "http://localhost:2024",
assistantId: "agent",
onCustomEvent: (event, options) => {
options.mutate((prev) => {
const ui = uiMessageReducer(prev.ui ?? [], event);
return { ...prev, ui };
});
},
});
```
Then you can push updates to the UI component by calling `ui.push()` / `push_ui_message()` with the same ID as the UI message you wish to update.
- UI for creating deployments and revisions
- APIs for managing deployments
- Observability data storage
- Operator/listener to reconcile deployments
- Agent Servers (agents/graphs)
- Backing services (Postgres, Redis, etc.)
For setup, refer to the [Hybrid setup guide](/langsmith/deploy-hybrid).
2. Click **+ Evaluator** > **Create from labeled data**
3. Enter a descriptive feedback key name (e.g. `correctness`, `hallucination`) | 1. **Projects** and select your project
2. Click **+ New** > **Evaluator** > **Create from labeled data**
3. Enter a descriptive feedback‑key name (e.g. `correctness`, `hallucination`) | | **Align an existing evaluator** | 1. **Datasets & Experiments** > select your dataset > **Evaluators** tab
2. In the **Align Evaluator with experiment data** box, click **Select Experiments** | 1. **Projects** > select your project > **Evaluators** tab
2. In the **Align Evaluator with experiment data** box, click **Select Experiments** | ## 1. Select experiments or runs Select one or more experiments (or runs) to send for human labeling. This will add runs to an [annotation queue](/langsmith/evaluation-concepts#annotation-queues).
langgraph dev"] --> B["Test Locallylanggraph dev"] --> C["Validatelanggraph up"] --> D["Deployvia UI or API"] style A fill:#1976d2,stroke:#0d47a1,stroke-width:2px,color:#fff style B fill:#1976d2,stroke:#0d47a1,stroke-width:2px,color:#fff style C fill:#f57c00,stroke:#e65100,stroke-width:2px,color:#fff style D fill:#388e3c,stroke:#1b5e20,stroke-width:2px,color:#fff ``` | Stage | Tool | Purpose | | -------------------------- | --------------------------------------------------------------------------------------------------------- | -------------------------------------------------- | | **Develop & Test Locally** | [`langgraph dev`](/langsmith/cli#dev) | Write and iterate on your graph with hot reloading | | **Validate** | [`langgraph up`](/langsmith/cli#up) | Test production-like behavior with full stack | | **Deploy** | [LangSmith UI](/langsmith/deployment-quickstart) or [Control Plane API](/langsmith/api-ref-control-plane) | Deploy to production with confidence | ### Recommended workflow 1. **Daily development**: Use `langgraph dev` for rapid iteration. 2. **Periodic validation**: Test major changes with `langgraph up`. 3. **Pre-deployment check**: Run `langgraph up --recreate` for a fresh build. 4. **Deploy**: Push to production via the [LangSmith UI](/langsmith/deployment-quickstart) or [Control Plane API](/langsmith/api-ref-control-plane). ## `langgraph dev` The [`langgraph dev`](/langsmith/cli#dev) command runs a lightweight server directly in your environment, designed for speed and convenience during active development. The key features include: * **No Docker required**: Runs directly in your environment. * **Hot reloading**: Automatically reloads when you change code. * **Fast startup**: Ready in seconds. * **Built-in [Debug Adapter Protocol](https://microsoft.github.io/debug-adapter-protocol/) support**: Attach your IDE debugger to the server for line-level breakpoints & debugging. * **Local storage**: State persisted to local directory.
If you use [`langgraph new`](/langsmith/cli) without specifying a template, you will be presented with an interactive menu that will allow you to choose from a list of available templates.
system | reasoning | user | assistant | tool
text | image | file | audio | video | tool\_call | server\_tool\_call | server\_tool\_result.
success | error.
id of a prior assistant message’s tool\_calls\[i] entry. Only valid when role is tool.
Use filters or sampling to control evaluator cost. For example, evaluate only threads under *N* turns or sample 10% of all threads. 5. **Configure an idle time**.
The first time you configure a thread level evaluator, you'll define the idle time — the amount of time after the last trace in a thread before it's considered complete and ready for evaluation. This value should reflect the expected length of user interactions in your app. It applies across all evaluators in the project.
Select the provider and model you want to use for your evaluator. Threads tend to get long, so you should use a model with a higher context window in order to avoid running into limits. For example, OpenAI's GPT-4.1 mini or Gemini 2.5 Flash are good options as they both have 1M+ token context windows. 7. **Configure your LLM-as-a-judge prompt.**
Define what you want to evaluate. This prompt will be used to evaluate the thread. You can also configure which parts of the `messages` list are passed to the evaluator to control the content it receives: * All messages: Send the full message list. * Human and AI pairs: Send only user and assistant messages (excluding system messages, tool calls, etc.). * First human and last AI: Send only the first user message and the last assistant reply. 8. **Set up your feedback configuration**.
Configure a name for the feedback key, the format for the feedback you want to collect and optionally enable reasoning on the feedback.
You can check when your evaluator was last run by heading to the **Evaluators** tab within a tracing project and clicking the **Logs** button for the evaluator you created to view its run history. **Inspect the data sent to the evaluator**
Inspect the data sent to the evaluator by heading to the **Evaluators** tab within a tracing project, clicking on the evaluator you created and clicking the **Evaluator traces** tab. In this tab, you can see the inputs passed into the LLM-as-a-judge evaluator. If your messages are not being passed in correctly, you will see blank values in the inputs. This can happen if your messages are not formatted in one of [the expected formats](/langsmith/online-evaluations-multi-turn#prerequisites). ***
• [Organization settings](#organization-settings): Org info and configuration
• [Workspaces](#workspaces): Workspace management
• [Organization members](#organization-members): Member management
• [Roles and permissions](#roles-and-permissions): Custom roles | **Core resources:**
• [Projects](#projects): Organize traces and runs
• [Runs](#runs): Individual execution traces
• [Datasets](#datasets): Test datasets for evaluation
• [Examples](#examples): Individual dataset examples
• [Experiments](#experiments): Comparative experiments | | **Security and authentication:**
• [SSO and authentication](#sso-and-authentication): Single sign-on setup
• [SCIM](#scim): Identity provisioning
• [Access policies](#access-policies): Attribute-based access control | **Monitoring and analysis:**
• [Rules](#rules): Automated run rules
• [Alerts](#alerts): Alert rules for monitoring
• [Feedback](#feedback): Scores and labels on outputs
• [Annotation Queues](#annotation-queues): Human review queues
• [Charts](#charts): Custom visualizations | | **Billing and accounts:**
• [Billing and payments](#billing-and-payments): Subscription management
• [API keys](#api-keys): Org-level keys | **Development and configuration:**
• [Prompts](#prompts): Prompt templates (LangChain Hub)
• [Deployments](#deployments): Deployment configurations
• [MCP Servers](#mcp-servers): Model Context Protocol servers | | **Analytics:**
• [Charts and dashboards](#organization-charts-and-dashboards): Org-level visualizations
• [Usage and analytics](#usage-and-analytics): Usage tracking and TTL settings | **Workspace management:**
• [Workspace settings](#workspace-settings-and-management): Members, settings
• [Tags](#tags): Metadata tagging system
• [Bulk Exports](#bulk-exports): Data export operations | **Additional information:** * [User-level operations](#user-level-operations): Operations for all authenticated users * [Permission inheritance](#permission-inheritance): How roles inherit across org/workspaces ## Legend * ✓ **Allowed**: User with this role can perform this action * ✗ **Not Allowed**: User with this role cannot perform this action * ⚠ **Partial**: User has limited access (see notes) ## Organization-level operations
(500m CPU, 1Gi per replica) | 1 (default) | 4 | 2 | 2 | 4 | | **Platform backend replicas**
(1 CPU, 2Gi per replica) | 3 (default) | 20 | 3 (default) | 3 (default) | 20 | | **Ingest queue replicas**
(1 CPU, 2Gi per replica) | 3 (default) | 24 | 3 (default) | 6 | 24 | | **Backend replicas**
(1 CPU, 2Gi per replica) | 2 (default) | 5 | 40 | 16 | 50 | | **Redis resources** | 8 Gi (default) | 26 Gi external | 8 Gi (default) | 13Gi external | 26 Gi external | | **ClickHouse resources** | 4 CPU
16 Gi (default) | 10 CPU
32Gi memory | 8 CPU
16 Gi per replica | 16 CPU
24Gi memory | 14 CPU
24 Gi per replica | | **ClickHouse setup** | Single instance | Single instance | 3-node
8 GB memory
10GB storage (external) | 2 CPU
8 GB memory
10GB storage (external) | 2 CPU
8 GB memory
10GB storage (external) | 2 CPU
8 GB memory
10GB storage (external) | 2 CPU
8 GB memory
10GB storage (external) | | **Blob storage** | Disabled | Enabled | Enabled | Enabled | Enabled | Below we go into more details about the read and write paths as well as provide a `values.yaml` snippet for you to start with for your self-hosted LangSmith instance. ## Trace ingestion (write path) Common usage that put load on the write path: * Ingesting traces via the Python or JavaScript LangSmith SDK * Ingesting traces via the `@traceable` wrapper * Submitting traces via the `/runs/multipart` endpoint Services that play a large role in trace ingestion: * Platform backend service: Receives initial request to ingest traces and places traces on a Redis queue * Redis cache: Used to queue traces that need to be persisted * Ingest queue service: Persists traces for querying * ClickHouse: Persistent storage used for traces When scaling up the write path (trace ingestion), it is helpful to monitor the four services/resources listed above. Here are some typical changes that can help increase performance of trace ingestion: * Give ClickHouse more resources (CPU and memory) if it is approaching resource limits. * Increase the number of platform-backend pods if ingest requests are taking long to respond. * Increase ingest queue service pod replicas if traces are not being processed from Redis fast enough. * Use a larger Redis cache if you notice that the current Redis instance is reaching resource limits. This could also be a reason why ingest requests take a long time. ## Trace querying (read path) Common usage that puts load on the read path: * Users on the frontend looking at tracing projects or individual traces * Scripts used to query for trace info * Hitting either the `/runs/query` or `/runs/
Via Okta Integration Network (recommended)
Via Custom App Integration