Eval Harness Starter Kit

What's in it

eval.py — ~150-line harness: runs your cases against any OpenAI-compatible chat endpoint (OpenAI, Anthropic compat, vLLM, Ollama, llama.cpp, OpenRouter), grades with contains / exact / regex / not_contains / judge (LLM-as-judge), prints a report, writes JSON, exits non-zero on failures — drop it straight into CI.
cases.example.yaml — five real starter cases: factual recall, format discipline, hallucination refusal, extraction, secret-leakage.
README.md — quick start + how to compare a frontier API vs this week's open-weights model with the same suite.

Quick start

unzip aiops-eval-kit.zip && cd aiops-eval-kit
pip install -r requirements.txt
cp cases.example.yaml cases.yaml        # edit with YOUR cases

export EVAL_BASE_URL=https://api.openai.com/v1
export EVAL_API_KEY=sk-...
export EVAL_MODEL=gpt-5.6-luna

python3 eval.py cases.yaml --json results.json

Why this matters

The week Ox Alpha dropped, the teams who could answer "should we switch?" in an afternoon were the ones with an eval suite already written. Cisco's numbers say 85% of enterprises are stuck piloting agents — reliability, not capability, is the blocker. A harness is the smallest piece of the boring reliability layer, and it compounds: every incident becomes a new case.

Pair it with the cost calculator to price the switch you just validated. New models covered weekly on This Week in AIOps.