This week, hundreds of AI engineers joined us at Interrupt NYC to hear how teams are taking domain-specific agents to production and to discover the latest LangSmith announcements.
Our latest releases focus on two pillars of the agent development lifecycle that help teams build with control and flexibility, compound the intelligence of their agents, and ensure strong governance: runtime and observability + evaluations.
Runtime gives agents what they need to operate in production (such as memory, channels, tools and secure execution environments). Observability and evals help teams understand behavior, measure quality, and improve agents they more they are used.
Sitting above is the intelligence layer, which works over the business logic, traces, evals and production data to accelerate this agent development loop.
Here’s what we shipped.
LangSmith Engine v2: Proactively detect agent issues and ship fixes faster
Improving agents is a multi-step process; a typical workflow requires engineers to spot issues, determine a root cause, propose a fix, test it, deploy it, and monitor for regressions.
We built LangSmith Engine as an in-platform agent that automates work at each step of the agent development lifecycle. Since launching in May, Engine has already helped engineers analyze more than 60M traces to diagnose tens of thousands of issues.
Engine v2 introduces major advancements for issue detection, testing and validation.
Proactively troubleshoot with Red Teaming
New Red Teaming capabilities help detect issues before they surface in production. Using production traces and repos, Engine can generate hypotheses about issues that may not have appeared in production yet, test those hypotheses, and surface confirmed failures for review.
Detect harder-to-spot issues
Engine now detects more issue types. In addition to errors and unmet user requests, it can identify performance trends across error rate, latency, and cost, as well as inefficient agent work such as repetitive tool calls or unnecessarily long trajectories.
Automatically test proposed fixes
For agents running on LangSmith Deployment, Engine can now validate proposed fixes prior to human review and deployment. Engine v2 runs the offending inputs against your agent to confirm the issue. Then it tests candidate fixes against a broader eval set until it finds one that resolves the issue. Once Engine has a satisfactory fix, it’s presented to the end user, where they can open a PR with a click for quick deployment.
The next release of self-hosted LangSmith will support BYOK for Engine.
Learn more about Engine v2.
Managed Deep Agents v0.8: New auth, memory, and channels
Managed Deep Agents combines the Deep Agents harness with managed infrastructure, creating the easiest way to go from business logic to an agent in production.
Our latest release addresses four challenges teams face when running agents in production: agent memory, authentication, channels and tool management.
Identity-scoped auth and memory
Managed Deep Agents already supported durable agent memory, and we’re excited to introduce new user-level memory so that agents can store context scoped to each authenticated user. This new layer provides a place to store caller-specific context that keeps it separate from shared agent-level memory.
The runtime does not copy content between layers, and access policies can be defined at each level to ensure that user-level information isn’t leaked into conversations.
Managed Deep Agents also now supports both agent and user-owned credentials for external connections (such as GitHub and Notion).
New and expanded channels
Agents work best when they operate where users already are. We expanded our Slack support to include file transfer, so users can send logs, spreadsheets, contracts, screenshots, and other files directly to an agent.
New HTTP channel support lets teams connect agents to any service that can send a JSON webhook. This is especially useful for customer-facing agents that live on multiple channels.
Built-in web search
Web search is one of the most common agent tools, so we decided to build it right into Managed Deep Agents.
Managed Deep Agents 0.8 brings web search as a prebuilt tool. Powered by Parallel, built-in web search gives agents access to current information without requiring teams to set up a separate vendor account, API key, or custom tool.
Learn more about Managed Deep Agents v0.8.
LangSmith Trajectories: Easily navigate agent sessions and trace data
Long-running agents can be hard to debug since a session can include many turns, tool calls, retries, and subagent handoffs. The full trace can get long and nested, and is often more information than you need to understand what happened.
That’s why we launched LangSmith Trajectories, a conversational view of an agent session. A trajectory aggregates messages from humans, AI, and tools across the main agent and any subagents, then shows them in the order chronologically. The result is faster debugging because teams can quickly see where behavior changed or broke.
Trajectories also make review workflows easier. SME reviews are critical for agent accuracy, but often, these reviewers aren’t the ones who built the agent. Trajectories give them a readable session view that they can score, flag, and annotate without having to parse execution metadata and other details.
Trajectories are a critical part of improvement loops. LangSmith online evaluators can score Trajectories, providing better input for judging agent behavior across a session.
Learn more about LangSmith Trajectories.
LangSmith Fine-Tuning: Open model post-training for specialized tasks
We built LangSmith Fine-Tuning to help teams get the benefits of fine-tuned open source models without needing to build the fine-tuning pipeline by hand. A model trained on your own data and intelligence can often perform specified tasks as well or better than a general purpose frontier model, and at a much lower cost and latency.
LangSmith Fine-Tuning enables teams to conduct supervised fine-tuning (SFT) to train an open model on high quality examples of a task. smithtune, the LangSmith Fine-Tuning CLI, helps turn trajectories into useful training data, handling the entire fine-tuning process:
- Building and preparing datasets from trajectories
- Training model with Baseten or Fireworks
- Evaluating results with LangSmith
- Serving tuned models and connecting them to applications
We benchmarked SmithTune internally to evaluate the quality of the workflow ourselves, and are incredibly excited about the results.
You can read the full breakdown here.
Custom Apps: Build custom interfaces around your agent data
Every team reviews agent data differently. LangSmith provides default interfaces for common workflows like trace review, experiment comparison, annotation, and evaluation analysis. But we heard from customers that they want more customized ways to interact with their data that align with their own workflows and quality criteria.
Custom Apps lets teams build, publish, and run custom interfaces on top of LangSmith data.
Instead of exporting data or maintaining a separate internal tool, teams can build against LangSmith APIs using templates, in-app chat, or coding agent guidance, and publish the app into their LangSmith workspace. Once published, the app becomes a shared interface for recurring workflows.
This is useful for annotation workflows where reviewers need a focused UI, experiment reviews where teams want to compare outputs across versions or slices, and trace review workflows where the best view depends on the application.
Learn more about Custom Apps in LangSmith here.
Owning your intelligence
Ultimately, our latest updates support our mission to enable every company to own their intelligence. In practice, this means that teams can build customized, domain-specific agents and applications tailored to the specific problems they want to tackle.
Watch the the Interrupt NYC keynote, featuring our CEO Harrison Chase, for a deeper dive into these announcements.
