Agents
OSWorld
A benchmark that runs multimodal agents inside real virtual machines and asks them to complete open-ended computer tasks across desktop apps, file operations, and multi-app workflows.
In depth
A benchmark that runs multimodal agents inside real virtual machines and asks them to complete open-ended computer tasks across desktop apps, file operations, and multi-app workflows. The original set has 369 tasks, each with a scripted starting state and an execution-based checker that inspects the actual result rather than the agent's description of it. At launch humans completed about 72% of tasks and the best model about 12%. It is now the headline number for computer-use models.
Primary source: arxiv.org
Example
In practice, developers reach for OSWorld when they need the capability described above as part of an AI feature or workflow.
Go deeper at Developers Digest
Hands-on guides, comparisons, and tutorials that cover Agents.
FAQ
Common questions
What is OSWorld?
A benchmark that runs multimodal agents inside real virtual machines and asks them to complete open-ended computer tasks across desktop apps, file operations, and multi-app workflows.
Why does OSWorld matter for AI developers?
OSWorld sits in the Agents part of the AI stack. Understanding it helps you make better decisions when building, debugging, and shipping AI features.
Where can I learn more about OSWorld?
Developers Digest publishes tutorials and videos that cover Agents topics including OSWorld. Check the blog and YouTube channel for hands-on walkthroughs.
Related
Related terms
Agents
Agent Handoffs
A multi-agent mechanism where one agent passes control of the conversation to a more specialized agent, rather than calling it as a subroutine.
Agents
METR Time Horizon
A capability metric from METR that describes what an AI system can do in human terms: the length of task, measured by how long skilled humans take to do it, that a model completes with 50% success.
Agents
tau-bench
A benchmark from Sierra that tests agents in multi-turn customer-service conversations, where a language-model-simulated user talks to an agent that has domain API tools and a written policy to follow.
Put this concept to work
Get Smarter About AI Dev
New tutorials, open-source projects, and deep dives on coding agents - delivered weekly.
One email per weekReal code, not theoryFree forever