EVALS

Langfuse is part of ClickHouse, Promptfoo is part of OpenAI, Galileo is part of Cisco, Traceloop is joining ServiceNow, Helicone joined Mintlify and Humanloop shut down. Who owns what in October 2026, the dates the vendors state, and a portability checklist built on OTLP ingest, licenses and self-hosting.

Evaluate AI agents in layers: deterministic tool checks, trajectory match, then an LLM judge. Working code for Pydantic Evals, Mastra and more.

Mastra in late 2026: goals, schedules, signals, AgentController GA, eval gates and MCP 2.x, with type-checked TypeScript examples and honest tradeoffs.

Pydantic AI V2 explained through v2.59.0: capabilities, typed output, Conversation storage, caching, graphs, evals, MCP Apps and Logfire, with checked code.

Cursor's latest agent-swarm experiment rebuilt a SQLite-like database from documentation and passed a held-out conformance suite. The bigger story is the shift from assigning code tasks to specifying, measuring, and governing a goal.

Harness Handbook maps coding-agent behavior to code evidence. Why code search falls short when one behavior spans prompts, tools, state and permissions.

SkillHone argues agent skills need decision history, not just the latest SKILL.md. Why saving the reasons behind each change helps future agents improve them.

Terminal-Bench and StateM show harness scaling now drives coding-agent scores. Why runbooks, durable state and recovery loops matter as much as model choice.

AI agent eval tools compared - Braintrust, Langfuse, Promptfoo, DeepEval, Ragas, and OpenAI Evals on offline vs online evals, LLM-as-judge, CI, and datasets.

Langfuse vs Braintrust vs Helicone, plus LangSmith and Arize Phoenix: 2026 pricing tiers, self-host options, agent-trace features, and OpenTelemetry support.

Dan Luu's agentic coding notes argue that coding agents compound only when the test, review, and task-selection loops are stronger than the code generator.

Image token compression cuts agent costs by rendering bulky context as images, but it is lossy. Where to use it, the byte-safety rule, and the evals it needs.

OpenAI Agent Builder, hosted Evals and reusable prompts shut down November 30. A migration plan to the Agents SDK, repo-owned prompts and eval receipts.

Agentic AI reliability is a systems problem. The Bayer and Thoughtworks PRINCE case study shows it comes from context routing, traces, evals and human review.

Agent evals need baseline receipts: Hex's data-agent lab compares candidates against stable production baselines, keeps evidence and judges changes by behavior.

Open-source LLM engineering platform: tracing, evals, prompt management, and datasets. Self-hostable, OpenTelemetry-native, with 50+ framework integrations.