Making an AI platform's agent behavior measurable.

The client had built a sophisticated multi-agent system handling customer support workflows. Each agent made decisions, called APIs, and processed documents autonomously. The problem? They had no idea what was actually happening. LangSmith captured traces, but they were just raw JSON dumps nobody looked at. When an agent misbehaved with wrong responses, infinite loops, or unexpected costs, the engineering team would spend 3-4 hours manually digging through logs to understand what went wrong. Their monthly AI spend was climbing unpredictably. They suspected certain agents were inefficient, but couldn't prove it. Leadership wanted cost attribution per customer, but the data wasn't structured for that.

We started with a 2-day discovery sprint to understand their agent architecture, existing data sources, and what questions they actually needed answered. From there, we designed a pipeline that would flow data from LangSmith into a queryable analytics layer. The key insight was that they didn't need real-time streaming for everything. Most questions could be answered with 15-minute latency. This let us build a simpler, more reliable batch pipeline rather than over-engineering with Kafka.

We built a three-layer architecture: The Data Ingestion layer uses a Python-based ETL pipeline that pulls trace data from LangSmith every 15 minutes, normalizes the nested JSON structures, and extracts key metrics including token counts, latencies, tool calls, and error rates. The Storage Layer is split by purpose: raw traces land in S3 for long-term retention and compliance, while aggregated metrics go into ClickHouse for fast analytical queries. This dual approach keeps storage costs low while enabling sub-second dashboard queries. For Visualization, Grafana dashboards provide real-time visibility into agent performance. Custom panels show token usage trends, error rates by agent type, latency distributions, and cost breakdowns by customer.

The pipeline runs on their existing AWS infrastructure with minimal additional resources. We used Lambda for the ETL jobs (cost-effective for their volume), S3 for raw storage, and a small ClickHouse cluster for analytics. We made several key technical decisions: We chose ClickHouse over Postgres because their query patterns, time-series aggregations and high cardinality metrics, are exactly what ClickHouse excels at. Queries that took 30+ seconds in Postgres now complete in under 1 second. We used S3 for raw data because compliance required 2-year retention. Storing raw JSON in S3 costs ~$5/month for their volume vs. $200+/month in a database. We went with Grafana over custom dashboards because it's battle-tested, self-hosted, and their team already knew it. No point reinventing the wheel.

Within two weeks of going live, the client identified two agents that were using 4x more tokens than necessary due to inefficient prompts. Fixing these saved them $3,000/month in API costs, more than the entire project cost. Debugging time dropped dramatically. When an agent started behaving unexpectedly, engineers could now open a dashboard, see exactly what happened (which tools were called, what the LLM returned, where it went wrong), and fix it in minutes instead of hours. Leadership finally got the cost attribution they needed. They could now see AI spend per customer, per agent, per day, enabling better pricing decisions and identifying their most expensive (and profitable) use cases.