AutoCua: a computer use agent that clicks by number, not pixel
In the README's demo clip, the web agent finishes a flight search in 6.1 seconds.
- Who
- Built and maintained by Ashish Yadav.
- When
- The repository opened on October 8, 2026; v0.1.5 on October 26 brought Linux.
- Where
- macOS, Windows 10/11, Linux with GNOME, Chrome and other browsers, and an iPhone or iPad on a Mac. Python 3.10+.
What was hard
Agents that work a screen often look at a screenshot and guess where to click. They miss a button by a few pixels or aim at something behind another window, and long tasks die at the context limit before they finish.
What it does about it
AutoCua gives the model numbers instead of coordinates. It merges the operating system's accessibility tree with OCR, stamps a number on each element, and when the model picks one, sends real mouse and keyboard events in at the OS input layer. A separate compression agent splices a summary into the transcript so long tasks keep going.
How it runs inside
- 1
You (main.py) → run_agent()
run_agent(mode=, provider=, model=, task=)One mode string decides whether it drives the desktop, a shell, the web or iOS.AutoCua/agent_launcher.py - 2
run_agent() → The computer use agent
The computer use loop startsThe parent agent owns the screen and can hand code work to the coder agent.AutoCua/mac/agent/computer_use/service.py - 3
Accessibility tree + OCR → The computer use agent
A numbered element tree and annotated screenshotAccessibility tree and OCR are merged, elements hidden behind other windows dropped, magenta numbers stamped on.AutoCua/mac/tree/element.py - 4
The computer use agent → Your model provider
Next action: one numberThe model never sees a coordinate; a number it was never shown is refused before anything moves.AutoCua/llm_provider/llm_manager.py - 5
The computer use agent → The apps on your screen
Click and type at the OS input layerOn macOS mouse events go in at kCGHIDEventTap, so apps get what your hand on the mouse would send.AutoCua/mac/controller/service.py
What goes in, what comes out
Is it for you
Good for
- Handing repetitive work that hops between apps to an agent
- Web tasks in your own Chrome, already logged in
- Python developers who want to swap model providers by changing two strings
Not for
- The shell tool is not a sandbox: no approval prompt before a command runs, and most of your home folder is reachable.
- It drives your real machine, browser profile and logins; the README calls that both the point and the risk.
- You need an API key for your model provider. Android is not available yet, and iOS mode needs a Mac.
- The Interception driver used for Windows UAC needs a paid licence from its author for commercial use.
Reviewed by Devalley’s editor before publishing. How the column is made