What's in it
- eval.py — ~150-line harness: runs your cases against any OpenAI-compatible chat endpoint (OpenAI, Anthropic compat, vLLM, Ollama, llama.cpp, OpenRouter), grades with
contains/exact/regex/not_contains/judge(LLM-as-judge), prints a report, writes JSON, exits non-zero on failures — drop it straight into CI. - cases.example.yaml — five real starter cases: factual recall, format discipline, hallucination refusal, extraction, secret-leakage.
- README.md — quick start + how to compare a frontier API vs this week's open-weights model with the same suite.
Quick start
unzip aiops-eval-kit.zip && cd aiops-eval-kit
pip install -r requirements.txt
cp cases.example.yaml cases.yaml # edit with YOUR cases
export EVAL_BASE_URL=https://api.openai.com/v1
export EVAL_API_KEY=sk-...
export EVAL_MODEL=gpt-5.6-luna
python3 eval.py cases.yaml --json results.jsonWhy this matters
The week Ox Alpha dropped, the teams who could answer "should we switch?" in an afternoon were the ones with an eval suite already written. Cisco's numbers say 85% of enterprises are stuck piloting agents — reliability, not capability, is the blocker. A harness is the smallest piece of the boring reliability layer, and it compounds: every incident becomes a new case.
Pair it with the cost calculator to price the switch you just validated. New models covered weekly on This Week in AIOps.
Get the field note by email
A short email when the calculators get new numbers, the directory gets new entries, and the week's episode lands. No hype — unsubscribe in one click.
Your address is used to send the field note and nothing else.
Who builds these tools
AIOps Field Notes is written by James Son, a network, security, and automation engineer, and everything here is tested in a live multi-vendor lab. That same lab work is behind two paid things for teams who run the infrastructure underneath the AI: