pyfsr.ai_eval¶
Repeatable, scored evaluation of FortiAI alert investigations (8.0.1+).
A suite defines stub MCP servers with planted ground truth (a SIEM, an EDR,
identity, CMDB, threat intel, plus distractors) and scenarios – an alert, the
tool calls a good investigation makes, the facts it should surface and the
right verdict. deploy() hosts the stubs on the appliance through the
bundled fsr-ai-eval-stub connector (a local listener, like the Teams
connector’s bot listener) and registers each as an MCP server;
run_suite() runs the investigations and scores each one from its trace.
CLI: pyfsr ai-eval deploy|run|log|teardown.