Loading page
About this tool
Decide what to trace, measure, log and redact for a production LLM agent, with a readiness score and sampling plan.
An agent observability checklist turns a vague 'add logging' ticket into a named list of spans, metrics, logs, evaluations and redaction rules, then scores how many of them your agent already emits. Signal names follow the OpenTelemetry semantic conventions for generative AI systems — the gen_ai.* namespace used by most tracing backends — so the plan maps onto whatever collector you already run. It is aimed at engineers taking an LLM agent from a working demo to something on call rotation can actually debug.
Open Agent Observability Checklist on AltFTool — it loads instantly in your browser.
Paste your code or data sample into the workspace.
Pick the format, conversion, or analysis you need.
Copy the polished result straight back into your project.
Weighted readiness score shows how far the required signals are from complete.
Turning on traits like tool use or regulated domain escalates the signals that actually matter.
Head rate, sampled runs per day and stored volume computed from your traffic and retention.
At minimum: one root span per run carrying a stable run id, a child span per model call with the requested and responding model ids, a child span per tool call, token usage split into input and output, operation duration as a histogram, and the prompt template version. Prompt and completion text should sit behind an explicit content-capture flag rather than being on by default.
OpenTelemetry publishes semantic conventions for generative AI systems that define the gen_ai.* attribute namespace — for example gen_ai.operation.name, gen_ai.request.model, gen_ai.response.model and gen_ai.tool.name — plus the instruments gen_ai.client.token.usage and gen_ai.client.operation.duration. Message content capture is opt-in, precisely because prompts often contain personal data.
Pick a daily trace budget first, then set the head rate to budget divided by runs per day: 20,000 traces a day out of 1,000,000 runs is a 2% head rate. Layer tail sampling on top that keeps 100% of errors, runs slower than your p95 target, guardrail blocks and negative feedback, since those are the runs you will actually open.
Redact inside a span processor or exporter so raw text never leaves your process, keep card numbers, government IDs and API keys out of telemetry entirely, hash identifiers you need for joins, and truncate IP addresses or round coordinates to about two decimal places (roughly 1.1 km). Set a short retention window on raw messages and make deletion automatic. This is informational guidance, not legal advice — confirm your obligations with a privacy professional.