What is computer use?
Computer use is a way for an AI model to operate a computer the way a person does: it looks at screenshots, then asks for mouse clicks, key presses and typed text. Anthropic's computer use tool defines those actions; your own code runs each one in a sandboxed desktop and sends back a fresh screenshot.
Part of Glossary: the terms behind MCP, coding agents and shared pages, defined plainly.
You send the model a task and the computer use tool in one API request. The tool gives it actions such as screenshot, left_click, type, scroll and zoom.
The model replies with tool calls: usually a screenshot first, then a click at a coordinate it read off that image.
Your code runs each call in a desktop you control, ideally a dedicated VM or container, and returns an image for screenshot and zoom or a short OK for the rest.
The model studies the new screenshot, decides the next step, and the loop repeats until it answers with text and no tool calls.
Coordinates are screenshot pixels, so if you shrink the image before sending it you scale the clicks back up before performing them.
Computer use versus a browser agent
Both let a model act instead of only answer. The difference is what the model can see. A browser agent gets the structure of a web page, so it can say "click the element with reference 12" and be right every time. A computer use agent gets pixels. It has to find the button in a screenshot, estimate where it is, and ask for a click at those coordinates. That makes it general, since any application that draws on a screen is fair game, and also slower and less precise. Anthropic's docs say it plainly: for tasks confined to web pages, use the browser tool instead.
Example: saving a file from a desktop app
You ask for a chart from a spreadsheet to be exported as a PNG. The model asks for a screenshot, sees the spreadsheet, and requests a click on the chart, then a key call for the export shortcut. Your code performs each action and sends back a new screenshot. A save dialog appears; the model types the filename and clicks Save. Then it asks for one more screenshot to confirm the dialog is gone and replies with text. Each loop costs a request, and each screenshot is roughly 1,000 to 1,800 input tokens, which is why long tasks prune old images from the history.
The reference implementation Anthropic publishes runs all of this inside a Docker container with a virtual X11 display, a lightweight desktop, Firefox and LibreOffice. That is the right mental model for production too: a throwaway machine with only what the task needs on it.
When to use it
Reach for computer use when the thing you need to automate has no API and is not a web page: a legacy desktop application, an installer, a native settings screen, a game. It is also useful for testing software the way a person experiences it, because it finds the problems a DOM-level test cannot, like a button that exists but is covered by something else.
Do not use it where a smaller tool fits. Reading a web page wants web fetch. Filling a form wants the browser tool. Calling a service wants an MCP server or a plain function. Computer use is the fallback when nothing structured is available.
Common mistakes
- Running it on your real desktop. The docs ask for a dedicated VM or container with minimal privileges, no access to account logins, and an allowlist of internet domains. Anything the model can see, it can be told to act on.
- Forgetting that screen content is input. Text in a document or an email can carry instructions, and the model may follow commands found in content even when they conflict with yours. Anthropic runs classifiers on screenshots to catch this, but keep a human confirming anything with real-world consequences.
- Sending full-resolution Retina screenshots. Oversized images are rejected, and a device pixel ratio of 2 means your coordinates are off by half unless you scale. The docs suggest 1024x768 or 1280x720 and warn against anything above 1920x1080.
- Trusting "done". The model sometimes acts as if an action succeeded without checking. Ask it to take a screenshot after each step.
FAQ
- Does computer use run on Anthropic's servers?
- No. The model only returns the action it wants, such as a click at a coordinate or a string to type. Your application executes it in a machine you provide and sends back the screenshot. Nothing on Anthropic's side ever connects to that machine.
- Is a computer use agent the same as a browser agent?
- A browser agent is the narrower case. It stays inside web pages and can read the page's element tree instead of guessing from pixels. A computer use agent sees the whole screen, so it can handle desktop apps and file dialogs, at the cost of working from screenshots and coordinates alone.
- Why does a computer use agent misclick?
- Small targets, heavily downscaled screenshots and vague instructions are the usual causes. The zoom action, keyboard shortcuts for fiddly widgets, and asking for a screenshot after each step so the model checks its own work all help.
Stop sending HTMLover Slack.
Create an account in a minute, connect your coding agent, and make your first board.