With observability, we can connect agent behavior to real business cases, catch performance bottlenecks and keep costs more predictable.

When we start with AI agent observability, we usually begin with the basics: capturing spans, tool calls and token counts, as covered in the previous observability post.

In production, observability is about more than raw events. It helps connect agent behavior to real business cases, catch performance bottlenecks and keep costs predictable.

When users interact with our agents, we need answers to real questions:

Which user waited 35 seconds for a response?
Why did the agent trigger four hotel searches instead of one?
Why did a single trip plan cost ten times more tokens than expected?
How do we trace internal database queries inside custom tools?

Today, we take our travel planner to the next level. We will learn how to inject business metadata with propagateAttributes(), diagnose reasoning loops, audit token leaks with wrapFunctionWithSpan(), instrument backend services with decorators, and configure production privacy guardrails.

Let’s make this work in a real project!

Getting Started

If you want code with me, I recommend you clone the project repository and move into its folder:

Check the available branches:

You will see two branches:

article-start: The base agent from our first article.
main: The completed agent with advanced observability.

To follow it step by step, switch to the starter branch:

But before starting, we need two API keys:

A Google AI Studio API key to run the Gemini model.
A Progress Agent Engineering integration key to capture traces and spans.

Create a .env file in the project root:

Verify that the agent runs:

Now, let’s inject business metadata into our traces.

In our previous article, the Progress Agent Engineering captured spans under the service name family-travel-planner. But in a busy production environment, your trace list receives hundreds of executions every minute.

Searching for a failed run by raw timestamp or random trace IDs like 4bf92f3577b3... is not easy, because when customer support receives a ticket, they hear: “User usr_4102 on the VIP tier tried to plan a trip to Ibiza, and the request timed out.”

The Progress SDK helps us solve this with propagateAttributes(). Think of this as a shared context for your execution. Any tags you inject automatically flow down to every child span created during that request, including Genkit flows, Gemini model calls and custom tool executions.

Open src/app.ts and wrap the trip planning call with propagateAttributes():

Now we do not need to pass user IDs through every function argument or manage low-level OpenTelemetry span objects.

Run the project from your terminal:

Now open the Progress observability dashboard at and go to Observe > Tracing:

Click on Filters in the top bar.
Select Tag, set condition to contains and type tier:vip (or user:usr_4102).
The dashboard filters out unrelated traffic and shows only matching runs.

Click on the execution, then select any child tool span like find-hotels:

The find-hotels span displays our business tags (user:usr_4102, tier:vip, destination:Ibiza). The tags flowed down the entire execution tree automatically.

So far, we have attached business context to our traces. Now, let’s see what happens when tools return unexpected data and agents get stuck in loops.

Diagnosing Reasoning Loops with Trace Visualizations

The AI agents operate in an iterative cycle: the model reads the prompt, calls a tool, reviews the tool’s output and decides what to do next.

What happens when a tool returns empty data?

Suppose a user asks for a luxury beachfront hotel in Ibiza for under €30 per night. Our hotel tool searches its records, finds nothing and returns an empty array: [].

To us, [] is a valid result. But to an LLM, an empty array is confusing. The model assumes its search parameters were slightly off, so it adjusts the keywords and queries again. It calls the tool a second time, gets [] again, and calls it a third and fourth time.

This pattern is called an agent reasoning loop. The agent repeats tool calls without making progress, wasting execution time and multiplying token costs.

When an agent loops, the trace visualization in Observe > Tracing reveals clear warning signs:

You can identify a reasoning loop by three clear signals:

Repeated tool spans: A long staircase of the same tool call.
Growing latency: Total duration jumps from 3 seconds to over 40 seconds.
Growing token usage: Each retry carries all previous failed attempts in the prompt history.

We fix this by combining two safety mechanisms:

Step limits (maxTurns): Set a hard limit on turns so Genkit stops the execution if the agent loops.
Defensive tool responses: Never return a silent empty array []. Return a clear message explaining why there are no results and instruct the model not to retry.

Open src/tools/hotels.ts and update the search logic with a defensive response schema:

Next, open src/agent.ts and set maxTurns: 4 on the generation call:

When the tool returns status: 'NO_MATCHES' with explicit instructions, the LLM stops immediately. Instead of looping, it tells the user to adjust their budget.

Run the project again. Open the execution trace in the Progress observability tool dashboard:

Look at the execution flow:

Turn 1 (Tool Gathering): Gemini analyzes the request and calls find-flights, find-hotels and find-activities.
Turn 2 (Synthesis): Gemini receives the tool outputs and builds the final itinerary.
Clean Execution: The flow finishes in two turns with eight total spans in 3.3 seconds. No loops, no retries.

We have protected our agent against reasoning loops. Now, let’s examine how heavy tool payloads cause token leaks and how to trace internal data operations.

Auditing Token Leaks and Tool Payload Sanitization

In tutorials, tools return clean, minimal JSON. In real applications, tools query partner APIs or databases.

A single hotel API call often returns 30 to 50 results. Each hotel object includes database IDs, tax breakdowns, room inventory codes, image URLs and cancellation policies.

If your tool passes that raw payload back to Genkit, the entire 20+ KB response enters the model context. Because agents send previous tool outputs on every turn, that bloated payload gets re-sent on every subsequent step.

The solution is use a Data Transfer Object (DTO) pattern: always filter external data so the model receives only the fields it needs.

Genkit tracks top-level tool calls automatically. But how do you trace internal operations, such as data pruning or local cache checks?

The Progress SDK provides wrapFunctionWithSpan() to create custom child spans inside any function.

Open src/tools/hotels.ts and add payload sanitization with wrapFunctionWithSpan():

When getHotelsForDestination() runs, it calls sanitizeHotelPayloadTraced() on the raw vendor data before returning clean results to the model.

Notice the impact on token consumption and performance:

Run the project again. In the Progress observability dashboard, expand the find-hotels tool span:

Notice the new child span:

Under find-hotels, a child span named sanitize-hotel-payload appears with type TASK.
The right panel displays the tag operation:payload-sanitization.
The span automatically inherits active business tags (user:usr_4102, tier:vip).

You can now measure the exact duration of your sanitization logic independently from external network latency.

So far, we have instrumented standalone functions. Now, let’s look at how to instrument class-based backend services with decorators.

Backend Services with @workflow and @task

What happens when our agent needs to rely on existing backend services, such as payment processors, reservation managers or inventory databases?

The Progress SDK provides TypeScript decorators (@workflow and @task) to connect class methods directly into the active trace tree.

Create src/services/booking.ts:

Make sure your tsconfig.json has "experimentalDecorators": true enabled.

By adding @workflow and @task, your service methods appear as child spans under the active execution. You can quickly see whether a delay came from the model or an internal database query.

So far, we have instrumented our application layers. Now, let’s configure production privacy guardrails.

Production Privacy Guardrails with traceContent

Observability requires capturing telemetry, but production systems must protect user privacy.

In local development, recording prompt texts, tool inputs and model outputs are helpful for debugging. In production, users enter sensitive information: traveler names, children’s ages and personal notes.

Under privacy regulations like GDPR, CCPA and SOC2, storing unmasked personal data in telemetry systems creates compliance risks.

The Progress SDK provides the traceContent option in bootstrap.ts to solve this cleanly.

Open bootstrap.ts and set traceContent based on your environment:

When NODE_ENV=production is set, traceContent: false tells the SDK to capture execution times, span hierarchies, tool names, token counts and costs, while keeping raw prompt text and personal data out of cloud logs.

Run the agent in production mode:

The terminal confirms that privacy mode is active:

Your trace records full operational metrics (latency, token counts and span hierarchy), while maintaining user privacy.

Conclusion

Observability is what helps us turn an AI experiment into a reliable, production-ready software system. Today, we explored five core observability practices with the Progress SDK:

Business Metadata: We used propagateAttributes() from the Progress Agent Engineering observability platform to attach user and session context across the entire span tree.
Loop Prevention: We combined maxTurns and defensive tool envelopes to stop agent reasoning loops.
Token Leak Auditing: We used DTOs to prune vendor payloads and tracked the sanitization with wrapFunctionWithSpan().
Backend Instrumentation: We used @workflow and @task decorators to trace service methods.
Production Privacy: We configured traceContent to protect sensitive user data in production.

With these practices in place, you can build AI agents with confidence that your system is inspectable, efficient and cost-effective.

Get started with the Progress Agent Engineering platform for free.

Try Now

Happy observability!