Evaluation Methodology
Paper trading and backtesting answer different engineering questions. One observes an agent in the unfolding present; the other moves an agent through historical chronology and exposes its complete decision and portfolio path.
Nine methodological differences explain why serious AI-agent development benefits from both approaches. BotTrade specializes in historical-market scenarios, sequential agent decisions, simulated execution, and inspectable risk evidence before live paper observation.
BotTrade lets a Python or MCP agent interact with a historical-market simulator, submit sequential decisions, receive simulated fills, and produce return and risk evidence. Scenario version, agent identity, trades, and results remain attached to the run. This makes backtesting useful for diagnosing how a policy behaved, not merely calculating an equity endpoint from a signal array.
A historical scenario can cover months of market chronology without waiting months for forward observations. That accelerates prompt, tool, memory, and allocation development. The speed does not make every result generalizable, so builders should inspect trades and use multiple BotTrade scenarios. Paper trading remains necessary for studying a system against the unfolding present and real operational time.
Backtesting lets builders deliberately choose episodes involving volatility, rotation, momentum, reversals, or event shocks relevant to the agent's intended behavior. BotTrade scenario metadata defines the universe, time range, cash, leverage, and benchmark. Report each scenario result separately so a selected stress test answers a named engineering question rather than becoming a universal performance claim.
Paper trading exposes the agent and builder to market events whose later outcome is genuinely unavailable. This tests source ingestion, response to ambiguity, and forward decision discipline. It also introduces changing data and operational conditions that complicate diagnosis. Preserve prompt, model, tool, and portfolio records with the paper run so a live-looking result still has an inspectable decision trail.
A long-running paper system reveals scheduling, process restarts, delayed data, provider outages, clock errors, duplicate actions, and state recovery problems. Historical BotTrade runs can test client interruption and resumable run state, but they complete on demand. Use paper trading to verify that the selected architecture remains operational when decisions must occur on an external schedule for weeks.
Give each BotTrade agent version documented model, prompt, tool, memory, and allocation provenance, then change the component implicated by run evidence. Inspect resulting decisions, trades, and risk rather than relying on a final score. The purpose is traceable attribution. Paper trading changes the surrounding market every day, making rapid component diagnosis slower and more dependent on longer observation.
BotTrade completed results include return, Sharpe, Sortino, maximum drawdown, trades, positions, and symbol-level evidence. Those measures expose portfolio path and activity that final equity alone hides. Paper systems should capture an equivalent record, including rejected and unfilled orders. Use the metric set to guide inspection, not to reduce a complex agent to one composite ranking.
Historical runs let teams discard malformed tools, incoherent prompts, unsafe sizing, and inactive policies before spending weeks on forward observation. Model calls and engineering time still have cost, so begin with BotTrade's sandbox smoke test and short scenarios. Promote configurations only after inspecting trades and risk. Paper trading should be reserved for versions with a defensible decision loop.
Use BotTrade first to build the complete agent loop, evaluate historical market structures, inspect failures, and select a small number of defensible configurations. Move those versions into paper trading with equivalent tracing to study operational continuity and new information. Feed observed failures back into another historical test or targeted simulator check before changing the forward-running policy.
BotTrade gives researchers historical scenarios, explicit mechanics, and complete decision records for examining models and architectures. Paper trading can then show how selected systems behave as genuinely new market information arrives and whether their operational processes remain reliable over time.