PluginPilot
PluginPilot shows whether an AI plugin really helps, and makes it better. Every plugin is measured against the same model without the plugin.
An AI plugin adds skills, tools and knowledge to an AI assistant, installed and updated as one package. The host is the assistant it runs in, such as Claude Code or Codex.
How it works
A plugin can sound helpful and still make answers worse. PluginPilot measures it directly: the same questions, the same model, once with the plugin and once without. It builds on Claude Code's own plugin evaluation and adds what is missing around it: flight plans with checked answers, the autopilot and the logbook.
A new plugin is built from a short written spec, with its facts checked against authoritative sources. An existing plugin needs only a flight plan.
Output: a plugin that installs and runs, and a flight plan of real questions with checked answers.
Every question runs with and without the plugin, on the same model and host, on fixed versions in a clean environment. Some questions are set aside as holdout cases, so later improvements are judged on questions nobody tuned for.
Output: a logbook record of what the plugin adds, where it fails and where it harms, kept whatever it says.
The autopilot reads why answers fail, tries changes of any kind, and keeps a change only if it beats the previous version, holdout cases included. It can never change the questions, answers or checks that judge it.
Output: an improved plugin, its evidence, and another test flight.
Why evidence
Most plugins are judged by the people who made them, on examples they picked.
That is flying by the view from the window. A plugin can pass its own demo and still give wrong answers, or worse answers than the assistant gives on its own. You only find out when someone relies on it.
PluginPilot flies on instruments the plugin does not control: answers checked against official sources, holdout cases the autopilot never sees, fixed model versions, and a record kept whatever it says. A gain that does not show on those instruments does not count.
A plugin's own test is a claim. The logbook is the evidence.
Examples
Helps navigate Berlin administration with information checked against official sources, starting with paperwork around the birth of a child.
Answers database questions by generating and executing read-only SQL. Results can be checked precisely, making it a strong test case for evaluation and improvement.
Turns ideas into clickable design prototypes, making interface exploration accessible beyond designers.
Contact
Have a plugin you want measured, or a question about the method?
Email PluginPilot