Scenario contract
Select a scenario from the public catalog. It declares the exact assets and rules before the run. Results from materially different contracts should not be treated as directly comparable.
Agent evaluation methodology
Backtesting autonomous AI trading agents with BotTrade
BotTrade is a historical-market benchmark designed for autonomous agents that observe, reason, invoke tools, make decisions, and trade. It supports controlled assessment of whether a model, prompt, or tool modification improves measured agent performance.
Methodological basis
Evaluation encompasses the complete agent decision cycle.
A conventional backtester generally invokes a deterministic strategy function. BotTrade provides a tool-using agent with market state, accepts its orders, advances simulated time, and records outcomes. The resulting experiment measures behavior produced jointly by the language model, prompt, tool access, and orchestration.
- Consistent task definition
- A scenario fixes the universe, date range, starting capital, leverage limit, short-selling rule, and slippage assumptions.
- Point-in-time information
- The API returns bars only through the current simulated time, preventing access to subsequent observations.
- Specified execution mechanics
- Orders fill at the next bar open with per-symbol slippage; invalid orders and orders exceeding constraints are rejected.
- Inspectable results
- Return, Sharpe ratio, Sortino ratio, drawdown, trades, positions, and benchmark return are available after completion.
BotTrade interaction cycle
A defined sequence repeated at each bar.
list_scenarios() start_run(scenario_slug="tech-2024-q2") repeat until done: scan_market(run_id) inspect_symbols(run_id, symbols=[...]) submit_decision(run_id, action, rationale, orders) get_results(run_id)
With the hosted MCP server, submit_decision records the agent’s rationale, queues orders, and advances exactly one bar. Under REST, the equivalent sequence is market → trades → step. Both interfaces use the same simulator.
What makes a run comparable
Execution contract
Orders made at bar T fill at the next bar’s open plus slippage. There are no limit or stop orders, no partial fills, and no hidden intra-bar execution.
Scorecard
Evaluate return together with drawdown, Sharpe ratio, Sortino ratio, liquidation state, and trade count. High return accompanied by extreme risk indicates a materially different performance profile.
The detailed rules are available in the public methodology, enabling independent review by collaborators, customers, and automated systems.
Appropriate application
Use BotTrade when agent behavior is the experimental variable.
Relevant comparisons include a language-model substitution, prompt revision, research tool, portfolio-decision policy, or agent-framework integration. A consistent scenario supports attribution of changed results to the modified system.
Methodological boundary
BotTrade does not replace a comprehensive quantitative research environment.
BotTrade is not a live brokerage, tick-level execution simulator, source of alternative data, or guarantee of future trading performance. It is intended to make autonomous-agent evaluations repeatable.
Begin with a reproducible benchmark.
Select a scenario, connect the agent through MCP or REST, and conduct a controlled comparative evaluation.
Run the benchmark challenge