Skip to main content

Computer-Use Browser Tools

A computer-use tool lets an agent drive a real web UI — navigate, click, type, scroll, screenshot — through a bounded, guarded, fully audited screenshot-step loop. Use it when the system your agent needs to operate has no clean API: an internal admin panel, a partner portal, a legacy back office. Endpoint paths below are relative; send them against https://api.orbit.devotel.io. Endpoint schemas are not restated here — see the agents endpoint reference for the full request and response bodies.
The console ships a dedicated Computer Use surface: open an agent and select the Computer use tab to filter its session list (All / Active / Completed / Failed) and replay each run’s per-step action trace. The tab observes and audits sessions — creating and updating tools, and dry-runs, stay on the API surface below. The console workflow is walked through in Operate governed computer-use agents from the console.

1. What a computer-use tool is

An agent’s tool belt today composes four tool kinds:
  • Custom tools — named HTTP endpoints you register once and reference from any agent.
  • MCP tools — capabilities exposed by an external MCP server.
  • A2A — skills another agent exposes through the A2A federation surface.
  • Computer-use tools — a bounded browser-operating loop that drives a web UI step by step.
For a computer-use tool, each agent turn proposes one browser action (navigate, click, type, …). Orbit evaluates that action against the tool’s guardrails, forwards approved actions to your sandbox executor, and writes an audit record for every step. The sandbox is a tenant-owned executor endpoint; Orbit never runs a browser itself. Containment is structural, not policy-level:
  • Bounded step budget — max_steps caps the loop (1–50, default 20). A run that exhausts its budget is forced to stop.
  • SSRF-pinned sandbox — the sandbox_url is validated at write time and re-pinned at every dispatch, so a tool can never be pointed at internal infrastructure.
  • Envelope-encrypted sandbox secret — sandbox_secret is stored envelope-encrypted and never returned by any read. Responses expose only a sandbox_secret_set flag.
  • Audit trail — every proposed action is recorded, whether it was allowed, denied, or gated on confirmation. A run can always be reconstructed later.

2. Create a computer-use tool

POST /api/v1/agents/{agentId}/computer-use-tools registers a tool on one agent. Each tool is tenant-scoped and lives under /api/v1/agents/{agentId}/computer-use-tools — full list / update / delete CRUD alongside create. Key fields on create: The browser action vocabulary is a closed set: navigate, click, double_click, right_click, type, key, scroll, move, drag, wait, screenshot, back, forward, refresh. navigate is the only URL-bearing action — it is the one the domain allowlist gates. Example:
The response is 201 with the created tool. Updates go through PATCH /api/v1/agents/{agentId}/computer-use-tools/{toolId} — re-sending sandbox_url re-validates it, sending sandbox_secret: null clears the stored secret, and omitting it leaves the current secret untouched. DELETE removes the tool. A 422 INVALID_SANDBOX_URL envelope names why the sandbox URL failed safety validation (protocol, blocked host, internal suffix, DNS, or private IP). Node.js:

3. Validate guardrails with a dry-run

POST /api/v1/agents/{agentId}/computer-use-tools/{toolId}/dry-run evaluates one proposed action against the tool’s guardrails before you let an agent loose on it. Send {"dispatch": false} to check the guardrail verdict only, or {"dispatch": true} (the default) to also forward the action to the sandbox exactly once — marked as a test so the sandbox treats it accordingly.
The response carries the guardrail verdict and, when dispatched, the sandbox result:
  • decision — allow, deny, or require_confirmation.
  • code — a machine-readable guardrail code on a non-allow verdict (see §5), and done: true from a live run when the step budget is exhausted.
  • result — the sandbox round-trip: HTTP status, duration, response body, and the error string when the sandbox rejected the action.
Dry-runs never consume an agent’s step budget and are safe to use in CI.

4. Audit every run

Every live run is reconstructable from the audit trail. Reads group the per-step action records by run — the same list and trace the dashboard Computer use tab renders, walked through in Operate governed computer-use agents from the console:
  • GET /api/v1/agents/{agentId}/computer-use/sessions — one row per run: status, first/last step timing, step count, and the operator-supplied task. Filter with ?status=all|active|completed|failed (default all) and ?limit=1–100 (default 25).
  • GET /api/v1/agents/{agentId}/computer-use/sessions/{sessionId} — one run’s full ordered action trace: each step’s action kind, guardrail decision, sandbox status, and duration.
A session’s status reflects how the run ended (completed, failed, cancelled) plus running / needs_input for runs still in flight. Step-by-step records are keyed by run, so the trace survives restarts and is the source of truth the dashboard session list renders.

5. Guardrail rules and runtime containment

Each proposed action passes through the tool’s guardrails before Orbit forwards it to the sandbox. The outcome is one of: At runtime the loop is additionally contained by the step_timeout_ms per-action timeout and by SSRF re-pinning on every single dispatch — the sandbox URL is re-validated per call, not once at creation. Access. Lists, session reads, and dry-runs require the agents:read scope; create / update / delete require agents:write plus the owner, admin, or developer role. All surfaces are tenant-scoped — an agent id outside your organization resolves to a 404.

Worked sequence — configure, dry-run, and audit in six languages

The programming loop for a computer-use agent is three calls: put the prompt template (with the tool instruction) on the agent, dry-run one proposed action to see the guardrail verdict before a live run, and read the run back from the session trail. The same loop in every supported language, in the reference’s fixed order.
The Python SDK is core-scope — it wraps the 8 core resources (messaging, voice, contacts, campaigns, verify, numbers) and reaches everything else through the generic client.request() escape hatch above. See the Python SDK. Verify the full six-language construction with pnpm --filter @devotel/docs run docs:regen before pushing — the generator re-renders the code tabs and the coverage audit reads the same fences, per Writing compliant API samples §4.

Reading a run back

The session read reconstructs one run from the per-step audit trail: the operator-supplied task, the derived status, and each step’s action kind, guardrail decision, sandbox status, and duration.
Token totals for the turns a run consumed arrive on that conversation’s execution trace; cost controls still apply to every step the loop takes.

Failure envelopes

The create and update calls fail a sandbox URL that misses safety validation with a 422 that names the reason (protocol, blocked host, internal suffix, DNS, or private IP):
Guardrail denies on a live step (ACTION_BLOCKED, DOMAIN_NOT_ALLOWED, STEP_BUDGET_EXHAUSTED) surface in the trace above, not as HTTP errors. For the wider agent failure classes — budget caps, provider errors, run-level faults — see Troubleshooting agent errors; full request and response bodies live in the Agents endpoint reference.

6. Where it fits with other tool kinds

Computer-use complements rather than replaces the other tool kinds:
  • Reach for a custom tool when the target system has an HTTP endpoint you can call.
  • Reach for MCP / A2A when the capability is already exposed by a service or another agent.
  • Reach for computer-use when the only interface is a rendered page — and accept the higher per-step cost in exchange for the closed action set, step budget, and full audit trail.