I built a CLI to test and eval MCP servers
12 hours ago
- #CLI
- #Testing
- #MCP
- MCPJam CLI performs MCP evals and End to End (E2E) testing by simulating user environments and testing workflows.
- Key benefits include discovering breaking workflows, benchmarking server performance, and programmatically testing queries.
- Setup requires creating `environment.json`, `tests.json`, and `llms.json` files for server connections, test configurations, and LLM API keys.
- The `environment.json` file configures MCP server connections, including OAuth tokens and server details.
- The `tests.json` file defines test cases, including queries, expected tool calls, and model configurations.
- The `llms.json` file stores API keys for providers like Anthropic, OpenAI, and OpenRouter.
- Commands like `mcpjam evals run` execute tests using the specified configuration files.
- CLI options include `--tests`, `--environment`, `--llms`, `--help`, and `--version`.