---
title: Evidence & Trust — CLI Agent Benchmark
url: https://arcana.otnelhq.com/docs/benchmark/evidence
---

Evidence & Trust — CLI Agent Benchmark 

  body { font-family: system-ui, sans-serif; max-width: 760px; margin: 40px auto; padding: 0 20px; color: #3D2B1F; background: #FFFBF5; line-height:1.7; }
  h1 { font-size: 24px; }
  h2 { font-size: 18px; margin-top: 32px; }
  code { background: #FFF5EC; padding: 2px 6px; border-radius:3px; font-size:13px; }
  pre { background: #FFF5EC; padding: 16px; border-radius:6px; overflow-x:auto; font-size:11px; }
  .hash { font-family: monospace; font-size: 12px; color: #6B5C4C; word-break: break-all; background: #FFF5EC; padding: 8px 12px; border-radius:4px; display:block; margin:8px 0; }
  ul { padding-left: 20px; }
  li { margin: 6px 0; }
  a { color: #D4859B; }
  table { border-collapse:collapse; width:100%; margin:16px 0; }
  th, td { border:1px solid #F0E0D0; padding:8px 12px; text-align:left; font-size:13px; }
  th { background:#FFF5EC; }

# Evidence & Trust

[← Back to benchmark](/benchmark.html)

## How to verify this benchmark is honest

### 1. Reproducible

Run `bash reproduce.sh` — same test file, same prompts, same verification. Anyone with the three CLIs installed can reproduce identical results.

### 2. Immutable Test File

SHA256: d697a6006ed360477c4b96d2d9eaf85880187e168029466c29b56ecb7ba56b5f

The test file was created BEFORE any agent ran. Hash proves no post-hoc modification.

### 3. Raw Implementations (SHA256)

Hermes: 21c699db843084d7bda37935a702be5c3d202a1b45a32dde45a4d436faf6cd9d

OpenCode: 3f25d9d8d80cb676a5be1eac44048306261b0fe7faf450e8a44586974a3d8272

Arcana: 7b969a14e43c2cdc7a7beef2a570e431f71e3fff50c998c8e4488ac860e5c0fd

### 4. Independent Verification

cd hermes && node semaphore.test.js # → 10/10 cd opencode && node semaphore.test.js # → 10/10 cd arcana && node semaphore.test.js # → 10/10 All pass independently. Not dependent on the agent that produced them.

### 5. Timing Methodology

- Measured at shell level via `date +%s%3N` — OS clock, not agent-reported
- Arcana: file-detection time (disk watch), not process exit time
- Hermes/OpenCode: process start to exit, including tool execution

### 6. What We Did NOT Do

| Anti-Pattern | Status |
| --- | --- |
| Cherry-pick runs | ❌ All were single-attempt |
| Modify test file after seeing results | ❌ SHA256 locked beforehand |
| Run agent with knowledge of solution | ❌ Clean subprocess each time |
| Use different test suites per agent | ❌ Identical semaphore.test.js |
| Hide failed attempts | ❌ There were none |
| Benchmark ourselves | ❌ Hermes ran as subprocess |

### 7. Agent Independence

- Hermes ran as subprocess (fresh `hermes chat`)
- OpenCode used its own auth system
- Arcana required account login + proxy key
- All three ran independently with no shared state

[← Back to benchmark](/benchmark.html)