A PHP application that evaluates capabilities (like programming or other textual generation) of different LLM models via the OpenRouter API.
LLModelScoring
A PHP toolkit that pits LLM models against your own tasks and scores them. It fetches every model from OpenRouter, runs a prompt you define, and evaluates the answers - with resumable runs and CLI or HTML reports.
What's in it
- A PHP 8.4 CLI app.
- An OpenRouter API client.
- A fetch-to-CSV workflow.
- A resumable test runner.
- An LLM-based content evaluator.
- CLI and HTML reporters.
- Pest tests.
The pipeline
Fetch pulls every model into a CSV you can edit. The test runner sends your `task.md` prompt to each enabled model, saves the raw responses, and can resume where it stopped. An evaluator LLM then scores the stored content, and a reporter renders CLI or HTML output. I split testing from evaluating so you can regrade stored answers without burning tokens again.