# Computer Use

## What it does

Controls a connected desktop directly. It observes the screen through accessibility APIs and screenshots, then clicks, types, and scrolls. Some actions depend on the desktop operating system. Platform-hosted web conversations default to the streamed Virtual desktop using accessibility elements and screenshots. Native desktop apps, including the Mac app connected to a platform-hosted assistant, use the connected computer unless explicitly directed to the virtual desktop. Explicit connected-computer targets override the web default. Each virtual desktop action uses the observation ID from its latest screenshot; browser actions or user handoff require a fresh observation. Virtual desktop control shares browser automation setup and user handoff. Use element IDs from the latest observation, or screenshot coordinates for controls not listed. AppleScript and window-only capture are unavailable.

## Setup required

None (built into the desktop app). The app will request the operating-system permissions it needs.

## Permissions

- Accessibility (mouse/keyboard control)
- Screen capture (seeing screen content)
- Each action is prompted individually for approval

## Common prompts

| You say...                                    | What happens                  |
| --------------------------------------------- | ----------------------------- |
| “Open my browser and go to my bank's website” | Opens app and navigates       |
| “Click the Submit button”                     | Clicks a specific UI element  |
| “Fill out this form with my info”             | Types into form fields        |
| “Take a screenshot of what's on screen”       | Captures current screen state |
| “Switch to Slack and check my DMs”            | Navigates between apps        |
| “Scroll down to the pricing section”          | Scrolls within an app         |

## Configuration

- No configuration needed
- Step limit of 50 actions per session
- Each action requires approval unless you create trust rules via the Allow button

## Tips & gotchas

- **Accessibility tree + screenshots.** The assistant reads the accessibility tree (same API screen readers use) on every step and takes a screenshot after each action, so it can see what the action did.
- **Element-based clicking.** It prefers clicking by element name rather than coordinates for reliability.
- **Coordinate scaling.** Click, scroll, and drag coordinates use screen points. Desktop screenshots include a conversion from image pixels when both dimensions are available. Window captures need element IDs or a fresh desktop observation, not scaling by the full display size.
- **Live editing.** The assistant uses app zoom controls and fresh screenshots to inspect small targets. If a cut depends on audio or an edit boundary it cannot observe, it asks for a timestamp, a positioned playhead, or source media.
- **Session caps.** Sessions are capped at 50 steps with loop detection.
- **Platform differences.** Dragging, opening apps by name, and AppleScript are available on macOS. Windows exposes only actions its desktop helper supports, so unsupported tools are not offered to the assistant.
- **Scripting first on macOS.** When an app can be driven by AppleScript, the assistant runs a script instead of moving your pointer. It asks the app's own scripting commands for what it wants where the app has them, falls back to clicking menus by script where it does not, and clicks and types only for what a script cannot reach.
- **Typing is not sending.** Asked to type or draft a message, the assistant types it and stops before pressing Enter. It sends only when asked to.
- **Your pointer comes back on macOS.** The pointer stays where the assistant last clicked while it works, so controls that only show on hover keep working. It returns to where you left it when the task ends, unless you have already moved it.
- **Batched actions on macOS.** When the assistant already knows several steps, such as opening a window, typing a URL, and pressing Enter, it can run them in one step and look at the screen once afterward. It stops at the first action that fails. This needs a desktop app that supports batched actions.
- **Single-window observations on macOS.** The observe tool accepts` capture_window_id`, a current native CGWindowID, not a browser tab or accessibility element ID. It captures only that window and its accessibility tree, even behind another app, without secondary windows or a desktop fallback. A compatible desktop app must explicitly advertise window-capture support; older or unsupported clients are rejected before capture.
- **Observation-only scope.** Window selection applies to one observe call, not the whole session: later click, type, and scroll actions still return normal desktop observations. Selection does not focus the window or restrict later input to it. Cropped screenshot coordinates are window-relative, not full-display coordinates; prefer accessibility element IDs and focus the intended window before acting.
- **Screen visibility.** Be mindful of what's visible on screen. Screenshots are sent to the AI model.