Oops! Something went wrong while submitting the form.
A computer use tool for an AI agent is the capability that lets the agent operate software through its screen, mouse and keyboard instead of through an API. In 2026 there are two ways to add one. You can give your agent low-level primitives (screenshot, click, type) from Anthropic, OpenAI or Google and build the environment and execution loop yourself, or you can give it a task-level operator: the Sai API from Simular, where one HTTP call hands a whole GUI task to a computer-use agent that already has a Windows or Linux computer, reports progress as events, and asks you for approval before anything sensitive. This guide explains both patterns, shows how the Sai computer use API works end to end, gives working integration code for the OpenAI Agents SDK, LangGraph, the Vercel AI SDK and MCP, and compares the four options on what you have to build, what they cost and how reliable they are.
Everything below was checked against the vendors' current documentation on 10 October 2026. Framework integration snippets for Sai are written against the public REST API and are marked for engineering review where Simular has not yet published an official example.
A computer use tool is a tool, in the function-calling sense, that gives a language-model agent the ability to see a computer screen and act on it: take a screenshot, move the cursor, click, scroll, type and press keys. With it, an agent can operate any application that has a graphical interface without an API, including desktop software, vendor portals and admin consoles that offer no integration at all. Without it, an agent can only act on the world through whatever APIs and code it has access to.
The term covers two quite different products. Model vendors ship computer use primitives: a tool definition the model understands (Anthropic's computer_toolset_20260801, OpenAI's computer tool, Gemini's computer_use) that returns actions for your code to execute against a display you provide. The second kind is a computer use agent as a tool: a hosted agent with its own computer that accepts a task description and returns a result, so your agent delegates the whole GUI job rather than driving each click. Sai is the example of the second kind in this guide.
Which one you want depends on whether you are building a computer-use agent or using one.
Primitives: your agent controls the mouse and keyboard. You add the vendor's tool to your model call. On each turn the model returns one or more actions (left_click at a coordinate, type a string, screenshot), your code executes them against a display you host (a Docker container with Xvfb, a Playwright browser, a VM), captures a new screenshot, and sends it back. You own the environment, the loop, coordinate scaling, error handling, safety checks and the bill for every screenshot. This is the right choice when the reasoning about the screen has to live inside your agent, for research, for custom UIs, or when you are building a computer-use product yourself.
Task-level operator: your agent delegates the job. You add one function tool to your agent, something like run_on_computer(task). When called, it posts the task to Sai, which provisions or wakes a cloud computer (or your own machine running the Sai desktop app), operates the screen with its own model stack, streams progress, pauses for approval when it needs to run a command or sign in, and returns the final text. Your agent never sees a screenshot. This is the right choice when the GUI work is a step in a larger workflow, when reliability per task matters more than control per click, or when you do not want to run desktop infrastructure.
The tasks that justify a computer use tool are the ones no API covers. Simular's own examples are a fair list: test a staging sign-up flow end to end in a real browser; pull this month's invoices from a vendor portal that only offers a web login; enter rows from a CSV into an internal form that has no import; follow a bug report step by step on a Windows machine and capture what happens; find inactive accounts in an admin console and export them. Each of these is a few minutes of clicking for a person and an unsolvable problem for an API-only agent.
The same rule tells you when not to reach for the tool. If a documented API or SDK exists, if the data is already local, or if you need exact byte-level output, write code. Computer use is slower and more expensive than a function call; its value is that it works where function calls cannot.
With primitives, reliability is whatever your loop achieves, and you only find out by building it. Sai publishes its numbers. On OSWorld 2.1, the long-horizon version of the standard computer-use benchmark, Simular reports Sai at a 77.3% partial score at $14.34 per task, on par with Claude Opus 5 and at a lower cost, with trajectories and methodology published. These are vendor-reported figures, but they are the only task-level completion and cost numbers published for any computer use tool in this comparison; the model vendors publish model scores, not product scores, and never a cost per task.
Cost per task also shows up in your own telemetry. Every /events response includes usage.costUsd and model-call counts for the current task and the session, so a framework tool can log what each delegated job cost without instrumenting a screenshot loop. And for repeated work, Sai's "run once, replay as code" model is the lever: after a task runs once, you can ask Sai to save the workflow as a named, parameterised skill; later runs replay the code and only query the model where the screen matters, with the agent taking over when a UI change breaks a step. Simular's neuro-symbolic computer use research reports learned policies cutting per-run cost by roughly 15 to 217 times and latency by 3.4 to 5.1 times on the benchmarks it tested, which is why a scheduled job that costs dollars the first time can cost cents thereafter.
A computer use tool is the most powerful tool you will give an agent, and every vendor says the same three things: run it in an isolated environment, treat what is on the screen as untrusted input (prompt injection tops the OWASP Top 10 for LLM applications), and put a human in front of consequential actions. Sai's API bakes the third into the protocol. Command execution raises an approval you must answer; sign-ins, password entry, Google account connections and phone verifications are link-only approvals that the API cannot complete on its own, so an agent can never be tricked into typing a credential. The task response lets you pre-approve a class of actions for one task without pre-approving everything, and POST /abort stops a task mid-flight.
Operational limits are published and worth designing around: 60 messages per hour per account, 1,200 event polls per hour (use wait=30, not tight loops), 20 approvals per minute, 30 uploads per hour at 25 MB each. The free plan's computer takes about two minutes to start on the first request of the day and has a daily time limit; paid plans get dedicated machines that hibernate when idle and wake on the next message.
The three vendor tools are excellent, and Sai can use their models underneath; the comparison is about what you take on when you adopt each.
Anthropic's computer use tool is the most mature primitive set: 17 actions including zoom, batched calls per turn, and detailed guidance on display sizes and coordinate scaling. The cost of adopting it is the environment. Anthropic's reference implementation is a Docker image running Xvfb, a window manager, Firefox and LibreOffice, and the tool definition alone costs about 4,500 input tokens per request before any screenshots. If you are building a computer-use product on Claude, start here; if you want Claude's models doing GUI work inside an existing agent, Sai lets you pick anthropic/claude-sonnet-5 or claude-opus-5-5 as the model and skip the environment.
OpenAI's computer tool reached GA in the Responses API with batched actions[] and a clean migration from computer-use-preview; the Agents SDK wraps it as ComputerTool with a safety-check callback. You still implement the Computer interface against Playwright or a VM, and OpenAI's own guidance is that for GPT-6 Astra it now recommends code execution first, with the computer tool "as an alternative".
Gemini Computer Use is the cheapest primitive to run and has the broadest environment list (browser, mobile, desktop) with normalised coordinates that make scaling trivial. It is still a Preview, injection detection is opt-in, and Google's own docs say to run it in a sandbox and away from critical tasks.
Sai is the only one of the four that is a complete computer-use agent behind an API. The flip side, and it is a real limitation, is that it does not expose screenshots or click primitives: if your agent needs to reason about pixels itself, Sai is not the tool. Its design bet is that most agent builders want the task done, not the screen.
The Sai API is a small REST surface under https://api.simular.ai/v1/agents, authenticated with a bearer key (Authorization: Bearer sapi_...). The free plan includes a pooled Windows cloud computer and daily credits; paid plans add dedicated Windows or Linux computers (up to five at once on the $50 Pay-as-you-go plan) and the option to run on your own Mac or Windows machine. Keys are created in the portal after a Google sign-in; free accounts verify a phone number first.
POST /v1/agents/message takes the task in plain language. Set wait: false to get a sessionId back immediately and poll for events (the default wait: true streams server-sent events instead). machineId is optional when you have one computer; model accepts auto or an id from /models; attachments carry files you uploaded first.
curl -s "$SAI_API_URL/v1/agents/message" \
-H "Authorization: Bearer $SAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"message": "Open Notepad, type hello, save it to the desktop as hello.txt",
"wait": false
}'
{ "sessionId": "cs_01HZY", "machineId": "free-3fA9kQ2xYz", "queued": false }
GET /v1/agents/events?sessionId=...&since=...&wait=30 long-polls for up to 60 seconds and returns the session status (running, needs_approval, idle or error), new events, an opaque cursor to pass back as since, the assistant text so far, and a usage object with token counts, model calls and costUsd for the task. Progress events (data-progress) are short narration lines like "Opening Notepad" with the tool in use; the API deliberately returns no screenshots, which is what keeps your agent's context small.
{
"events": [
{ "type": "data-progress", "data": { "text": "Opening Notepad", "tool": "computer" } }
],
"cursor": "1790000012345:3",
"status": "running",
"text": ""
}
When Sai wants to run a command, take an action you should see first, or needs a choice made, the session status becomes needs_approval and the response carries an approval object. You answer with POST /v1/agents/approve and a response of yes, no, or task (approve this and similar requests for the rest of the task). Choice approvals take a selections array. Some approvals are link-only: connecting a Google account, typing a site password, a phone verification. The API cannot answer those; it gives you an approvalUrl to hand to the human who owns the account, and the task resumes once they complete it. This is the part most hand-built computer-use loops never get right, and it is a first-class event here.
curl -s "$SAI_API_URL/v1/agents/approve" \
-H "Authorization: Bearer $SAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{ "approvalId": "ap_7Qw", "response": "yes" }'
When status is idle, text holds Sai's answer. Follow-up messages to the same sessionId continue the conversation and the desktop state; POST /new-session starts fresh and POST /abort stops a running task. Files go up through POST /upload (raw bytes, 25 MB, PDF, images, CSV, code) and are referenced as attachments. A complete Python loop that starts a task, prints progress, auto-approves command prompts and returns the text is about 60 lines; Simular publishes it in the quickstart and examples, and the version below is the one we wrap into framework tools in the next section.
Create a free API key, send one POST /v1/agents/message, and your agent has a computer. Wrap it as a function tool in the OpenAI Agents SDK, LangGraph or the Vercel AI SDK, or add the MCP server to Claude Code, Codex or Cursor. The free plan includes a cloud computer; the $50 Pay-as-you-go plan adds dedicated Windows or Linux machines, five in parallel.