Public eval methodology

"Does an agent actually work on Crank" should be an answerable, checkable question -- not a claim. This page documents how we measure it: a reference agent, run in paper mode, over a fixed window, against point-in-time market data, with a full cost model and a strategy-consistency score alongside the raw outcome. The harness is a script in the repo (scripts/eval/run_reference_eval.py); anyone can run it and get the same numbers.

Measured outcomes, not promises

Every number on this page is a measured outcome over one fixed eval window, on one reference strategy, in paper mode. It is not a return, a profit projection, or an expectation of future performance (see Fee schedule and the platform's language rules). Simulated behaviour over a single historical window does not indicate what any agent will do going forward.

What "reference agent" means here

The harness runs a minimal, fully-specified implementation of the same loop shape the Crank portfolio loop skill teaches for portfolio_loop: observe -> assess -> plan -> simulate -> execute -> verify -> journal. For this eval it is an SMA-crossover long/flat strategy on a single asset -- deliberately simple, so every step in the report is auditable by hand. It is not a "best" strategy; it is a reference implementation that exercises the full loop and the full cost model.

Eval discipline (why the harness is built this way)

Three findings from 2025-26 live-trading-agent eval literature (LiveTradeBench, PortBench, CLQT) shaped the harness:

1. Point-in-time data only -- no look-ahead. At step i the agent only ever sees OHLCV bars [0..i]. The eval loop in run_reference_eval.run_reference_eval is structured so a later bar is never indexed before its own step -- there is no code path that can leak a future price into a decision.
2. Strategy-consistency scoring alongside outcome. A good-looking outcome that came from the agent ignoring its own stated plan is not evidence the loop works. Every step records both the planned intent (from assess/plan) and the executed action (after any guardrail veto -- the reference run uses a position-flip cooldown guardrail), and strategies.backtest.score_strategy_consistency reports the match rate. A mismatch is not a failure of the harness; it is the guardrail working and the harness being honest about it.
3. Full cost modeling. Every paper fill is charged the real cost stack an agent would pay live: the platform's technology service fee (see Fee schedule), a slippage haircut, and a Solana priority-fee estimate. strategies.backtest.compute_trade_cost computes this per fill; nothing here nets fees out of the reported outcome.

Running it

# Reproducible, offline, no API key -- runs against the bundled frozen fixture.
python3 scripts/eval/run_reference_eval.py

# Replay against a freshly recorded pull instead of the frozen fixture
# (needs Django settings configured + BIRDEYE_API_KEY).
python3 scripts/eval/run_reference_eval.py --data-source live \
    --window-start 2026-06-01T00:00:00Z --window-end 2026-06-06T00:00:00Z

# Write the full JSON report (per-step planned/executed log, trade-by-trade
# cost breakdown, equity curve) alongside the printed summary.
python3 scripts/eval/run_reference_eval.py --out /tmp/eval_report.json

The default (fixture) mode is the reproducibility guarantee: same inputs, same seeded reference data, same deterministic strategy -> byte-identical output every run, on any machine, with no network access. --live-fees swaps the hand-built LOCKED fee-schedule snapshot for a live read of the admin-updatable FeeShareConfig row, for checking the harness against current governance parameters rather than the schedule at time of writing.

Reference run (fixed window)

Window: SOL/USDC, 1h bars, 2026-06-01T00:00Z .. 2026-06-06T00:00Z (120 bars), reference SMA(8)/SMA(24) crossover, $10,000 starting paper capital, 4-bar flip cooldown guardrail, scripts/eval/run_reference_eval.py default (fixture) mode:

Harness check	Measured value
Bars evaluated, point-in-time (no look-ahead possible)	120
Trades executed	4
Strategy-consistency score	1.0 (120/120 steps, 0 mismatches)
Cost stack charged on every fill (fee + slippage + priority)	$354.8881 total, itemized per trade in the JSON report

The headline output of the harness is the audit trail, not a performance number: the full equity curve, ending equity, per-step planned-vs-executed log, and trade-by-trade cost breakdown are all in the JSON report (--out /tmp/eval_report.json), reproducible byte-for-byte by anyone. The measured outcome over the window is reported there unvarnished, whatever its sign -- the fixture window is deliberately a flat, trendless stretch (buy-and-hold over the same 120 bars moved -0.12%), the kind of regime that exercises cost accounting and guardrail vetoes rather than flattering the reference strategy.

This is one run, on one fixed window, on one simple reference strategy. It is published so the harness itself -- point-in-time discipline, cost accounting, consistency scoring -- is checkable, not as a claim about what any live agent will earn. Re-run the command above to reproduce these exact numbers, or point --data-source live at a different window to measure a different one.

Third-party eval venues

Recall Network runs verifiable paper-trading arenas with a public leaderboard -- infrastructure that overlaps with, rather than competes against, this eval harness. No Crank-branded agent has been entered on any external leaderboard. Entering one would be a public representation of the Crank brand, so any such entry is a deliberate decision made by Engaij, Inc. -- never an automatic side effect of this harness.

Whatever Engaij, Inc. decides about external venues, the methodology on this page does not change:

• Complementary, not competitive. A third-party verifiable venue does not replace this page's self-published, reproducible-from-a-script methodology.
• The reference strategy is an audit fixture, not a flagship. The SMA-crossover reference above is intentionally minimal so every step is auditable by hand. It is not what Engaij Labs would present publicly as a representative Crank strategy.
• Language rules apply everywhere. Any external venue's profile and leaderboard copy is reviewed by Engaij Labs against the platform's language rules (no returns or performance-promise framing) before Crank appears there, because that copy is the venue's UI, not Crank's.

Recall Network competition assessment

Recommendation: GO, at low priority, effort small-to-medium (1-2 dev-days). Three decisions sit with the project maintainers, not with the harness, before any public entry happens:

1. Which strategy represents Crank publicly. Not the SMA-crossover audit fixture above -- see scripts/eval/run_reference_eval.py --strategy composite, a regime-aware entrant built on the same declarative rule-tree evaluator the live composite executor and MCP compose tools use (strategies.composite.evaluate_definition) plus the real regime detector the backtest strategy adapters use (strategies.regime.RegimeDetector, ADX / SMA slope / ATR percentile / volume). It is a candidate, not a foregone choice -- a different strategy may be preferred entirely.
2. Recall UI/branding compatibility with language rules 5-8. Any profile or leaderboard copy Recall renders for the entry needs the same review as any other Crank-branded surface before it goes live (no "returns"/ "investment"/performance-promise framing, per hard rules 5-8).
3. Timing. Independent of (1) and (2) -- when to actually enter, relative to other roadmap priorities (currently low priority).

Status (R12b): items independent of the maintainer decisions above are built. (a) The composite/regime-aware second reference strategy exists and runs offline/reproducibly, same as the SMA fixture -- run_composite_reference_eval in scripts/eval/run_reference_eval.py. (b) A Recall-submission-format adapter exists -- to_recall_submission_format/--emit-recall-format -- documented at the time as an unverified best-effort mapping pending a look at Recall's real API.

Recall API docs reviewed (R12c): read against Recall's published docs (docs.recall.network -- Endpoints, Trading guide, Quickstart, "your-first-trade") on 2026-08-26. Finding: the adapter's field-name guesses do not match Recall's actual submission model, and the adapter's whole shape is the wrong shape. Recall competitions are not a batch "submit your backtest result" API -- there is no endpoint that accepts a pre-computed fills-array + portfolio-time-series JSON like to_recall_submission_format produces. Instead an agent competes by calling Recall's trading simulator live, per-trade, during the competition window:

• POST /api/trade/execute (sandbox https://api.sandbox.competitions.recall.network, production https://api.competitions.recall.network), Authorization: Bearer <team API key>.
• Request body is a token swap, not a side/price/notional fill: {"fromToken": "<token address>", "toToken": "<token address>", "amount": "<string quantity>", "reason": "<optional string>"}, with optional fromChain/toChain for cross-chain routing (EVM chains + Solana/SVM are supported).
• Response echoes an executed transaction object (id, timestamp, fromToken/toToken, fromAmount/toAmount, price, teamId, competitionId, chain fields) -- Recall computes portfolio value and leaderboard rank server-side from these live calls, not from a submitted summary.
• Live standings: GET /agent/portfolio (total return, Sharpe ratio, max drawdown, volatility). Registration is a separate profile/agent-creation step (not yet reviewed in detail) that issues the API key trade/execute needs.

Net effect: to_recall_submission_format's fills/portfolio-series/ consistency-score shape is a useful internal audit-trail export (it is still exactly what this harness's paper-run result looks like), but it is not a Recall submission payload and cannot become one by fixing field names -- turning this harness into a live Recall entrant means driving the reference strategy's plan_fn decisions through real-time trade/execute calls during an actual competition window, an integration shape (a live loop against a real venue, real API key, real per-trade auth) genuinely different from this harness's offline/reproducible/no-network design. That work, and the entry decision itself, remain gated on decision point 1 above plus maintainer sign-off -- this revision only corrects the record on what Recall's API actually requires; it does not add a network call.

Status: the corrected API understanding above is the recorded next step, but it is explicitly not scheduled as implementation work yet -- decision point 1 (which strategy represents Crank publicly) blocks decision point 2 (branding review) and blocks all implementation work: registering a Crank team/agent profile with Recall for an API key, building the live per-trade trade/execute loop, and any live/paper competition entry itself. Decision point 3 (timing) remains an independent, separately open maintainer decision, as noted above. No code changed in this revision; nothing here is authorized to run against Recall's sandbox or production endpoints until the maintainer decisions above are made.