LLM Observability Starter

LLM Observability Starter is a self-hosted collector for teams that need visibility into LLM calls across applications, gateways, RAG services, and agent runtimes.

It records traces, latency, token usage, cost, fallback, errors, user feedback, and evaluation results through a small HTTP API. The component can run independently, or pair with an LLM gateway as an event sink.

What Is Included

Area	Included
Collector API	POST /v1/events/llm-call, POST /v1/events/feedback, POST /v1/events/eval-result
Query API	GET /v1/usage, GET /v1/traces/{trace_id}, GET /v1/export/llm-calls.csv
Runtime	FastAPI service with Docker Compose deployment
Storage	SQLite persistence for local and self-hosted deployment
SDK	Lightweight Python emitter client
Agent handoff	AGENT.md and CONTRACT.yaml for coding-agent integration

Repository

The source code is hosted on GitHub:

github.com/supernature885/ai-radar-llm-observability-starter

Quick Start

cp .env.example .env
docker compose up --build

Record one LLM call:

curl http://localhost:8010/v1/events/llm-call \
  -H "Authorization: Bearer local-observe-key" \
  -H "Content-Type: application/json" \
  -d '{
    "trace_id": "trace-demo",
    "request_id": "req-demo",
    "app": "example-app",
    "provider": "openai",
    "model": "gpt-4o-mini",
    "status": "success",
    "latency_ms": 820,
    "input_tokens": 1200,
    "output_tokens": 180,
    "cost_usd": 0.0012
  }'