Run it yourself

Test your own agent

The benchmark runs on Inspect, an open-source tool for testing AI models, so it works with any model Inspect supports.

1Install

pip install "agentic-commerce-trust-bench[openai,anthropic] @ git+https://github.com/roe-ai/actb"

2Run a model

inspect eval agentic_commerce_trust_bench/agentic_commerce_trust --model openai/gpt-5.6-terra

Add a judge to measure whether the agent warned the user:

inspect eval agentic_commerce_trust_bench/agentic_commerce_trust \
  --model anthropic/claude-opus-5-5 -T judge=openai/gpt-5.6-luna

3Look at the results

inspect view

Options

Pass with -T name=value.

judge: An AI model that decides whether the agent warned the user. Off by default.
images: How many photos the agent sees when it opens a listing. The default, 2, matches the leaderboard; 0 shows text only.
dataset: Another task file in the same format, or a Hugging Face dataset ID.
message_limit: The most messages per task. The default, 40, is about 10 turns.

What the code measures

Single-listing tasks are direct and multiple-listing tasks search; problem listings are traps and good ones controls.

bought, per kind of task: how often the agent bought the listing in question. Lower is better for problem listings, higher for good ones.
opened_first: of the tasks where the agent bought something, how often it opened the listing first.
warned_rate and silently_bought (need judge): how often the agent told the user about the problem, and how often it bought a problem listing without a word.

Get on the leaderboard

Leaderboard runs use all 1,000 tasks, the provider’s default settings, two photos per listing and no web access. Open an issue on GitHub with your Inspect logs to submit a model or agent.

Cite

@misc{roeai2026actb,
  title  = {Agentic Commerce Trust Benchmark},
  author = {{Roe AI}},
  year   = {2026},
  url    = {https://github.com/roe-ai/actb},
  note   = {Version 0.1.0}
}