Task Time Budgets
Each task controls a distinct champion, with a 5–20 minute execution budget. Gameplay continues during model reasoning and action execution.
LOLPlay: Benchmarking Real-TimeVisual Agents in League of Legends
36Champion-specific tasks
82Subgoal checks
6Task families
324Task evaluations
25.0%Best task success
Abstract
Visual agents must coordinate reasoning and action when the environment continues to evolve during inference. We introduce LOLPlay, a real-time visual-agent benchmark in League of Legends. Its 36 champion-specific tasks combine resource management, combat, cooperation, and objectives through screenshots and mouse–keyboard actions. Each task includes automated initialization and executable state/event evaluation, with 82 subgoal checks distinguishing partial progress from completion. Across 324 task evaluations of three models at three reasoning-effort settings, the highest task success rate is 25.0%, shared by GPT-6 Astra at low and medium effort; medium effort also achieves the highest subgoal success rate of 40.3%. High effort improves neither metric over low effort for any model. Analyses reveal incomplete subgoal composition and actions executing after the observed scene changes, motivating agents that integrate visual reasoning with timely control.
Each point is one model–effort configuration. Lines connect low → medium → high effort.
| Model | Effort | Latency | Task success |
|---|
MODELS
Task Statistics
Task Family Distribution
The 36 tasks span six primary control demands. Click a family to inspect its tasks and executable outcome checks.
Executable Evaluation
Task completion is verified independently from the agent's final message. A successful task satisfies every required condition, including its event order and deadline.
| Evidence | What is checked | Examples |
|---|---|---|
| Final state | Player state after termination | Level, inventory, skill ranks, attribute growth |
| Deadline state | A snapshot at the prescribed endpoint | Health, mana, survival at the deadline |
| Recorded events | Event identity, timestamp, and required order | Personal kills, assists, dragons, and turrets |
Challenge Phenomena
Familiar desktop inputs must be coordinated with a changing world. The benchmark exposes several recurring demands on real-time visual agents.
| Phenomenon | Control demand |
|---|---|
| Observation–action delayReasoning continues while the scene evolves. | Refresh observations and replan when enemies, combat, or interfaces change. |
| Subgoal compositionPartial progress does not guarantee completion. | Satisfy all resource, skill, survival, and objective conditions together. |
| Deadline controlSome outcomes must hold at a specific endpoint. | Maintain the required state until the task's time budget expires. |
| Ordered objectivesCorrect outcomes can require a particular sequence. | Personally kill a Voidgrub before destroying the target turret. |
| Visual groundingThe agent receives no structured unit-state feed. | Read the HUD, localize targets, allocate abilities, and navigate the shop from pixels. |
Human feasibility check. Two players, each with more than 200 prior League of Legends matches, cross-validated the full suite under the same configurations, budgets, and evaluator. All 36 tasks were completed. This verifies task feasibility; it is not a population-level estimate of human performance.
Leaderboard
36 tasks / configuration
| Rank | Model | Approach & Details | Turns |
|---|
1 Task success requires all task conditions. Subgoal success gives equal weight to tasks by averaging each task's fraction of completed checks.
2 324 task evaluations: 3 models × 3 effort settings × 36 tasks. Each configuration contributes one recorded attempt per task. Initialization failures remain in the denominator and receive no success credit; comparisons are descriptive.
3 Results use the current instruction-aligned 82-check suite. The revision re-evaluates 54 existing trajectories without new gameplay rollouts.
Updated: 25 October 2026
Key Findings
Reasoning Effort and Task Completion
Case Studies
Two archived trajectories show the gap between timely local actions and verified task completion.
Trajectory Showcase
Benchmark Resources
Complete Task Catalog36 tasks · 82 checks
Champion-specific objectives, time budgets, and the executable outcome checks.
Explore tasks ↗ JSON ↓Evaluation Results9 configurations
Task and subgoal success, request latency, and responses per executed episode.
JSON ↓Citation
Cite the LOLPlay paper: arXiv:2611.03685.
@misc{yu2026lolplaybenchmarkingrealtimevisual,
title={LOLPlay: Benchmarking Real-Time Visual Agents in League of Legends},
author={Zhaojian Yu},
year={2026},
eprint={2611.03685},
archivePrefix={arXiv},
primaryClass={cs.AI},
url={https://arxiv.org/abs/2611.03685},
}