Run bash reproduce.sh — same test file, same prompts, same verification. Anyone with the three CLIs installed can reproduce identical results.
SHA256: d697a6006ed360477c4b96d2d9eaf85880187e168029466c29b56ecb7ba56b5f
The test file was created BEFORE any agent ran. Hash proves no post-hoc modification.
Hermes: 21c699db843084d7bda37935a702be5c3d202a1b45a32dde45a4d436faf6cd9d
OpenCode: 3f25d9d8d80cb676a5be1eac44048306261b0fe7faf450e8a44586974a3d8272
Arcana: 7b969a14e43c2cdc7a7beef2a570e431f71e3fff50c998c8e4488ac860e5c0fd
cd hermes && node semaphore.test.js # → 10/10 cd opencode && node semaphore.test.js # → 10/10 cd arcana && node semaphore.test.js # → 10/10
All pass independently. Not dependent on the agent that produced them.
date +%s%3N — OS clock, not agent-reported| Anti-Pattern | Status |
|---|---|
| Cherry-pick runs | ❌ All were single-attempt |
| Modify test file after seeing results | ❌ SHA256 locked beforehand |
| Run agent with knowledge of solution | ❌ Clean subprocess each time |
| Use different test suites per agent | ❌ Identical semaphore.test.js |
| Hide failed attempts | ❌ There were none |
| Benchmark ourselves | ❌ Hermes ran as subprocess |
hermes chat)---
title: Evidence & Trust — CLI Agent Benchmark
url: https://arcana.otnelhq.com/docs/benchmark/evidence
---
Evidence & Trust — CLI Agent Benchmark
body { font-family: system-ui, sans-serif; max-width: 760px; margin: 40px auto; padding: 0 20px; color: #3D2B1F; background: #FFFBF5; line-height:1.7; }
h1 { font-size: 24px; }
h2 { font-size: 18px; margin-top: 32px; }
code { background: #FFF5EC; padding: 2px 6px; border-radius:3px; font-size:13px; }
pre { background: #FFF5EC; padding: 16px; border-radius:6px; overflow-x:auto; font-size:11px; }
.hash { font-family: monospace; font-size: 12px; color: #6B5C4C; word-break: break-all; background: #FFF5EC; padding: 8px 12px; border-radius:4px; display:block; margin:8px 0; }
ul { padding-left: 20px; }
li { margin: 6px 0; }
a { color: #D4859B; }
table { border-collapse:collapse; width:100%; margin:16px 0; }
th, td { border:1px solid #F0E0D0; padding:8px 12px; text-align:left; font-size:13px; }
th { background:#FFF5EC; }
# Evidence & Trust
[← Back to benchmark](/benchmark.html)
## How to verify this benchmark is honest
### 1. Reproducible
Run `bash reproduce.sh` — same test file, same prompts, same verification. Anyone with the three CLIs installed can reproduce identical results.
### 2. Immutable Test File
SHA256: d697a6006ed360477c4b96d2d9eaf85880187e168029466c29b56ecb7ba56b5f
The test file was created BEFORE any agent ran. Hash proves no post-hoc modification.
### 3. Raw Implementations (SHA256)
Hermes: 21c699db843084d7bda37935a702be5c3d202a1b45a32dde45a4d436faf6cd9d
OpenCode: 3f25d9d8d80cb676a5be1eac44048306261b0fe7faf450e8a44586974a3d8272
Arcana: 7b969a14e43c2cdc7a7beef2a570e431f71e3fff50c998c8e4488ac860e5c0fd
### 4. Independent Verification
cd hermes && node semaphore.test.js # → 10/10 cd opencode && node semaphore.test.js # → 10/10 cd arcana && node semaphore.test.js # → 10/10 All pass independently. Not dependent on the agent that produced them.
### 5. Timing Methodology
- Measured at shell level via `date +%s%3N` — OS clock, not agent-reported
- Arcana: file-detection time (disk watch), not process exit time
- Hermes/OpenCode: process start to exit, including tool execution
### 6. What We Did NOT Do
| Anti-Pattern | Status |
| --- | --- |
| Cherry-pick runs | ❌ All were single-attempt |
| Modify test file after seeing results | ❌ SHA256 locked beforehand |
| Run agent with knowledge of solution | ❌ Clean subprocess each time |
| Use different test suites per agent | ❌ Identical semaphore.test.js |
| Hide failed attempts | ❌ There were none |
| Benchmark ourselves | ❌ Hermes ran as subprocess |
### 7. Agent Independence
- Hermes ran as subprocess (fresh `hermes chat`)
- OpenCode used its own auth system
- Arcana required account login + proxy key
- All three ran independently with no shared state
[← Back to benchmark](/benchmark.html)