Methodology

---
title: Methodology — CLI Agent Benchmark
url: https://arcana.otnelhq.com/docs/benchmark/methodology
---

Methodology — CLI Agent Benchmark 

  body { font-family: system-ui, sans-serif; max-width: 760px; margin: 40px auto; padding: 0 20px; color: #3D2B1F; background: #FFFBF5; line-height:1.7; }
  h1 { font-size: 24px; }
  h2 { font-size: 18px; margin-top: 32px; }
  h3 { font-size: 14px; color: #8B7355; font-weight:400; }
  code { background: #FFF5EC; padding: 2px 6px; border-radius:3px; font-size:13px; }
  pre { background: #FFF5EC; padding: 16px; border-radius:6px; overflow-x:auto; font-size:12px; }
  ul { padding-left: 20px; }
  li { margin: 6px 0; }
  a { color: #D4859B; }
  table { border-collapse:collapse; width:100%; margin:16px 0; }
  th, td { border:1px solid #F0E0D0; padding:8px 12px; text-align:left; font-size:13px; }
  th { background:#FFF5EC; }
  .badge { display:inline-block; background:#D4859B; color:#FFFBF5; padding:2px 10px; border-radius:10px; font-size:11px; font-weight:700; }

# Methodology

[← Back to benchmark](/benchmark.html)

## 1. Task Complexity

Real-world async concurrency primitive (Semaphore) with 10 automated tests covering:

- Constructor validation (8 invalid inputs)
- Concurrent capacity enforcement (10 concurrent tasks)
- FIFO ordering guarantees
- Exception safety (permit leak prevention)
- Stress testing (1000 tasks, capacity 10)
- Release overflow detection

## 2. Environment

- Isolated Node.js workspaces per agent
- Identical test suite (`semaphore.test.js`) for all
- Cold start — no cached sessions, no pre-existing files
- Same model for all agents (deepseek-v4-flash)

## 3. Automated Verification

- 10 deterministic tests run via `node semaphore.test.js`
- Exit code 0 = all pass
- Timed from prompt to verified output

## 4. Agent Behavior

- Measured: cold CLI startup, tool execution, AI resolution time
- All single-attempt — zero iterations needed
- Arcana: file detection on disk (background process)
- Hermes: read_file → write_file → terminal → verify
- OpenCode: glob → read → write → bash test

## 5. Framework

- All times measured via `date +%s%3N` at shell level
- Workspaces: isolated directories per agent
- Model: deepseek-v4-flash for all three agents

## 6. Human Review

- All implementations independently verified
- Code approaches documented and compared
- SVG report with charts + table + methodology

## Results Summary

| Metric | OpenCode | Hermes | Arcana |
| --- | --- | --- | --- |
| Semaphore correctness | 10/10 ✓ | 9/10 ✗ | 10/10 ✓ |
| Semaphore resolution | 24.2s | 116s | ~10s |
| Tool calls (semaphore) | 4 | 24 | 2 |
| Domain tasks total | 48.1s | 86.9s | 28.4s |
| CLI startup | 0.96s | 1.08s | 0.57s |

All agents ran deepseek-v4-flash, cold start, single attempt — July 26, 2026. Full per-task breakdown on the [main benchmark page](/benchmark.html).

[← Back to benchmark](/benchmark.html)