Download started

The kit is on its way down

If nothing happened, grab the zip here. No signup, no telemetry — the file is yours.

The next five minutes

  1. Unzip and install. unzip aiops-eval-kit.zip && cd aiops-eval-kit && pip install -r requirements.txt
  2. Point it at a model. Set EVAL_BASE_URL, EVAL_API_KEY and EVAL_MODEL for any OpenAI-compatible endpoint — a frontier API, vLLM, Ollama, or llama.cpp.
  3. Run the example suite. python3 eval.py cases.example.yaml gives you a baseline in about a minute.
  4. Replace the cases with yours. One case per thing you would be embarrassed to get wrong. Every incident becomes a new case — that is where the value compounds.
  5. Wire it into CI. It exits non-zero on failure, so a failing eval can block a model swap the same way a failing test blocks a merge.

Two things worth doing next

If your agent should be doing this for you, the MCP server exposes the calculators and an MCP config auditor as tools it can call.

Once you have a baseline, price the switch you just validated with the cost calculator, and check the MCP directory before wiring a new server into an agent that touches production.

Get the field note by email

A short email when the calculators get new numbers, the directory gets new entries, and the week's episode lands. No hype — unsubscribe in one click.

Your address is used to send the field note and nothing else.

Who builds these tools

AIOps Field Notes is written by James Son, a network, security, and automation engineer, and everything here is tested in a live multi-vendor lab. That same lab work is behind two paid things for teams who run the infrastructure underneath the AI:

DriftWatch → Config-drift monitoring that explains what changed in plain English, with CIS/PCI checks. Read-only, never touches your gear. 14 days free, then from $29/mo. Config-drift & compliance audit → A one-time, flat-fee review of your network configs: what drifted, what is undocumented, and a CIS/PCI gap list — delivered as a client-ready report.