OSReward: Cross-platform evaluation for computer-use reward models¶
Executable evaluation protocol, not official human-labeled trajectories or OS-Shepherd training. Local benchmark:
osreward-mini.
Paper information¶
| Field | Details |
|---|---|
| Paper | Paper or original source |
| Institution | The University of Hong Kong / Nanjing University / NUS / USTC / Xi’an Jiaotong University / University of Oxford / Fudan University |
| First public release | 2026-07-30 |
| Author code | Project page: code, benchmark, data and weights. |
| Local adapter or method key | os-shepherd |
| Local reproduction code | src/auto_research/agent_research/osreward.py |
Paper method and evidence¶
Accuracy can hide a judge that labels nearly every computer-use trajectory successful. OSReward uses human-verified Windows, macOS, Ubuntu and Android tasks/trajectories, plus Hard and Multi subsets. It reports separate success/failure recall, their mean balanced accuracy, and a leniency audit. OS-Shepherd-100K trains open 9B and 35B judges.
flowchart LR
A[Computer-use trace] --> B[Completion evidence]
B[Completion evidence] --> C[Success and failure recall]
C[Success and failure recall] --> D[Balanced accuracy and leniency audit]Original paper figure¶
Figure 1 shows the evaluation and reward model framework. From the original paper; copyright remains with the authors.
Core equation¶
Reported results¶
Official OSReward has 1,019 trajectories. The strongest closed judge has about 89.7% accuracy; many judges show leniency bias. OS-Shepherd-9B scores 86.1% accuracy and 86.3% balanced accuracy, reportedly 30–60× cheaper than a frontier commercial judge. No production A/B.
Local reproduction¶
osreward-mini spans four platforms and task types. It distinguishes UI activity, proof of completion and unmet requirements rather than trusting the agent's success claim. In 120 deterministic episodes at seed 42 both recalls and balanced accuracy are 1.0 and leniency is zero. Fixed-seed metrics. This validates protocol behavior, not learned judge generalization; no official screenshots or 9B/35B training.