clickthrough/skill/SKILL.md

---
name: clickthrough-http-control
description: Control a local computer through the Clickthrough HTTP server using screenshot grids, OCR, zoomed grids, and pointer/keyboard actions. Use when an agent must operate GUI apps by repeatedly capturing the screen, reading visible text, refining target coordinates, and executing precise interactions (click/right-click/double-click/scroll/type/hotkey) with verification.
---

# Clickthrough HTTP Control

Use a strict observe-decide-act-verify loop.

## Getting a computer instance (user-owned setup)

The **user/operator** is responsible for provisioning and exposing the target machine.
The agent should not assume it can self-install this stack.

### What the user must do

1. Install dependencies and run Clickthrough on the target computer (default bind: `127.0.0.1:8123`).
2. Expose access path to the agent (LAN/Tailscale/reverse proxy) and provide the base URL.
3. Configure secrets on target machine:
   - `CLICKTHROUGH_TOKEN` for general API auth
   - `CLICKTHROUGH_EXEC_SECRET` for `/exec` calls
4. Share connection details with the agent through a secure channel:
   - `base_url`
   - `x-clickthrough-token`
   - `x-clickthrough-exec-secret` (only when `/exec` is needed)

### What the agent should do

1. Validate connection with `GET /health` using provided headers.
2. Refuse `/exec` attempts when exec secret is missing/invalid.
3. Ask user for missing setup inputs instead of guessing infrastructure.

## What the agent can actually see

The agent does **not** inherently see the remote desktop.
Clickthrough provides screenshots, OCR data, window metadata, and input endpoints — not native live vision.

That means:
- `GET /screen` and `POST /zoom` return image data the agent may need to inspect explicitly
- `POST /ocr` returns machine-readable text blocks when text extraction is enough
- the OpenClaw `image` tool is the right fallback when the agent needs judgment about visual layout, icons, button styling, dialog structure, or other non-OCR cues
- every visual conclusion is only as fresh as the last screenshot; after an action, recapture before assuming the UI changed as expected

Do not write or think as if the agent is directly watching the screen in real time.
Say what you actually have: screenshots, OCR output, and fresh verification captures.

## Mini API map

- `GET /health` → server status + safety flags
- `GET /displays` → detected displays in zero-based API order
- `GET /screen?screen=0` → full screenshot (JSON with base64 by default, or raw image with `asImage=true`)
- `POST /zoom?screen=0` → cropped screenshot around point/region (also supports `asImage=true`)
- `GET /windows` → discover visible desktop windows and their handles/processes
- `POST /windows/action` → focus/restore/minimize/maximize/close a matched window
- `POST /launch` → start an app/process without dropping to a shell
- `POST /wait?screen=0` → wait for text, window, or visual state changes
- `POST /vision/diff?screen=0` → compare screenshots or regions for meaningful visual change
- `POST /vision/stability?screen=0` → measure short-interval visual stability
- `POST /ocr` → text extraction with bounding boxes from full screen, region, or provided image bytes
- `POST /ocr/find?screen=0` → search OCR output for matching text candidates
- `POST /action?screen=0` → single interaction (`move`, `click`, `scroll`, `type`, `hotkey`, ...)
- `POST /action/verify?screen=0` → execute one action plus structured success verification
- `POST /batch?screen=0` → sequential action list
- `POST /exec` → PowerShell/Bash/CMD command execution (requires configured exec secret + header)

### Display selection

- Use `GET /displays` before operating on multi-monitor systems.
- Use `?screen=X` on `/screen`, `/zoom`, `/ocr`, `/action`, and `/batch`; invalid values fall back to `screen=0`.
- Treat returned `region` and OCR bounding boxes as global desktop coordinates, not screen-local coordinates.
- Do not assume `screen=1` starts at `(0,0)`; it may start at `(1920,0)`, `(-1920,0)`, or another global offset.
- If a screenshot came from `/screen?screen=1`, keep using that response's `region` metadata when forming later `/action` targets.
- Window rectangles from `GET /windows` are also in global desktop coordinates. Use them to sanity-check which monitor the app is really on before clicking.

### OCR usage

- Prefer `POST /ocr` when targeting text-heavy UI (menus, labels, buttons, dialogs).
- Use `mode=screen` for discovery, then `mode=region` for precision and speed.
- Use `language_hint` when known (for example `eng`) to improve consistency.
- Filter noise with `min_confidence` (start around `0.4` and tune per app).
- Treat OCR as one signal, not the only signal, before high-impact clicks.
- Current response shape is nested under `result.blocks`, not top-level `blocks`. Parse the real payload before assuming the endpoint failed.
- OCR can be noisy on dense shopping pages, streaming apps, and button-heavy sidebars. Re-crop tightly before escalating.

### Screenshot + `image` tool usage

Use the OpenClaw `image` tool when OCR is not enough.
This is especially useful for:
- identifying which visible button looks like the primary confirm action
- understanding dialog layout or pane structure
- distinguishing similar nearby controls by icon, spacing, or emphasis
- checking whether a visual state changed after a click
- telling you where something is and where to click when text alone is not reliable

Good pattern:
1. capture with `GET /screen` or `POST /zoom`
2. hand that screenshot to the `image` tool
3. ask a precise question about the visible UI
4. when click targeting matters, ask the model to describe **where the target is** or provide an approximate click point inside the crop
5. convert the answer into a concrete Clickthrough target
6. act once
7. recapture and verify again, or use `POST /action/verify` when the action+postcondition loop is simple enough to bundle cleanly

Prefer vision over guessing.
If OCR is fragmented, partial, or ambiguous, stop inferring and ask the vision model where the control is.
The model should help answer things like:
- which visible button is the real primary action
- whether the target is left/right/top/bottom within the crop
- which of several similar buttons is the one to click
- an approximate click point inside the provided image bounds

Ask narrow questions.
Good:
- "Which button in this dialog is the primary confirmation action?"
- "Is the scan still running, or does this look complete?"
- "Which of these tabs appears selected?"
- "Where is the orange Buy Now button in this 620x890 crop? Return one x,y coordinate inside the image bounds."
- "Which visible control says Stop Recording, and where should I click?"

Bad:
- "What should I click?"
- "Use your eyes and do the task"
- anything that assumes the model has live continuity without a new screenshot
- requesting coordinates without telling the model the image bounds or expected output format

### Header requirements

- Always send `x-clickthrough-token` when token auth is enabled.
- For `/exec`, also send `x-clickthrough-exec-secret`.

## `POST /action` request shape (important)

`/action` always expects an `action` plus an optional `target` object.
Do **not** invent top-level `x` / `y` fields.

Minimal pixel click:

```json
{
  "action": "click",
  "target": {"mode": "pixel", "x": 100, "y": 200},
  "button": "left",
  "clicks": 1
}
```

Minimal grid click:

```json
{
  "action": "click",
  "target": {
    "mode": "grid",
    "region_x": 0,
    "region_y": 0,
    "region_width": 1920,
    "region_height": 1080,
    "rows": 12,
    "cols": 12,
    "row": 6,
    "col": 8,
    "dx": 0.0,
    "dy": 0.0
  }
}
```

Other canonical examples:

```json
{"action": "move", "target": {"mode": "pixel", "x": 100, "y": 200}}
{"action": "double_click", "target": {"mode": "pixel", "x": 100, "y": 200}}
{"action": "right_click", "target": {"mode": "pixel", "x": 100, "y": 200}}
{"action": "scroll", "target": {"mode": "pixel", "x": 100, "y": 200}, "scroll_amount": -500}
{"action": "type", "text": "hello world", "interval_ms": 20}
{"action": "hotkey", "keys": ["ctrl", "l"]}
```

Rules:
- `dx` / `dy` belong inside `target`, not beside it.
- `type` and `hotkey` usually do not need a `target`.
- For pixel targets, `x` / `y` are global desktop coordinates.
- For grid targets, copy the exact `region_*`, `rows`, and `cols` basis from the screenshot/zoom you actually used.

## When to use `/exec`

Prefer structured GUI control first:
- `/screen`, `/zoom`, `/ocr` to observe
- `/action` or `/batch` to interact

Use `/exec` only when it is the cleanest available tool for the job, for example:
- querying machine state that the GUI does not expose well
- performing an explicit user-requested shell/system task
- recovering from a blocked GUI flow when normal interaction failed

Prefer `GET /windows`, `POST /windows/action`, and `POST /launch` for app lifecycle tasks before falling back to `/exec`.
Avoid using `/exec` for routine in-app clicks, menu navigation, or text entry when the GUI can be driven directly.
When a task can be completed with window focus/restore, keyboard shortcuts, screenshots, OCR, and normal actions, stay out of `/exec` entirely.

## Core workflow (mandatory)

1. Call `GET /windows` first when the task mentions a known app; focus/restore the right window before screen hunting.
2. Call `GET /screen?screen=0` with coarse grid (e.g., 12x12), or another selected display.
3. Identify likely target region and compute an initial confidence score.
4. If confidence < 0.85, call `POST /zoom` with denser grid (e.g., 20x20) and re-evaluate.
5. **Before any click**, verify target identity (OCR text/icon/location consistency).
6. If OCR is insufficient, inspect the screenshot explicitly with the OpenClaw `image` tool instead of pretending you can already see enough.
7. Execute one minimal action via `POST /action`.
8. Re-capture with `GET /screen` or use `POST /wait`, `POST /vision/diff`, `POST /vision/stability`, or `POST /action/verify` to verify the expected state change.
9. Repeat until objective is complete.

## Verify-before-click rules

- Never click if target identity is ambiguous.
- Require at least two matching signals before click.
- Good signal pairs include:
  - OCR text + expected UI region
  - OCR text + matching button shape/icon nearby
  - dialog title text + expected button position within that dialog
  - known app/window focus + expected control location
  - OCR candidate + vision-model localization inside the same crop
- If confidence is low, do not "test click"; zoom and re-localize first.
- If OCR and layout disagree, trust neither blindly; recrop and ask vision a narrower localization question.
- For high-impact actions (close/delete/send/purchase), use two-phase flow:
  1) preview intended coordinate + reason
  2) execute only after explicit confirmation.

## Precision rules

- Prefer grid targets first, then use `dx/dy` for subcell precision.
- Keep `dx/dy` in `[-1,1]`; start at `0,0` and only offset when needed.
- Use zoom before guessing offsets.
- Avoid stale coordinates: re-capture before action if UI moved/scrolled.

## Safety rules

- Respect `dry_run` and `allowed_region` restrictions from `/health`.
- Respect `/exec` security requirements (`CLICKTHROUGH_EXEC_SECRET` + `x-clickthrough-exec-secret`).
- Avoid destructive shortcuts unless explicitly requested.
- Send one action at a time unless deterministic; then use `/batch`.

## Reliability rules

- After every meaningful action, verify with a fresh screenshot.
- On mismatch, do not spam clicks: zoom, re-localize, and retry once.
- Prefer short, reversible actions over long macros.
- If two retries fail, switch strategy (hotkey/window focus/search) instead of repeating the same click.

## Fallback ladder for uncertain targeting

1. Full-screen capture with a coarse grid.
2. Zoom into the candidate area with a denser grid.
3. OCR the full screen or the tighter region.
4. Re-anchor on a more reliable nearby control, title, or label.
5. Try a keyboard-first flow if the app supports it.
6. Use `/exec` only if GUI control is blocked and shell-level intervention is genuinely cleaner.

Do not skip from "uncertain click" straight to random retries.

## Concrete screenshot -> `image` -> action example

Example loop:
1. `GET /screen?screen=0` to capture the current app state
2. if the UI is text-heavy, try `POST /ocr` first
3. if OCR does not answer the real question, pass the screenshot to the OpenClaw `image` tool with a narrow prompt like:
   - "In this save dialog, which visible button is the primary action?"
   - "Is there a dismiss/close button in the top-right of this modal?"
4. map the answer back to a Clickthrough target using the returned grid/region metadata
5. click once with `POST /action`
6. recapture the screen
7. optionally use `POST /wait` or another `image`/OCR check to confirm the result

The key rule is simple: screenshot first, interpret second, click third, verify fourth.
Do not collapse those steps into fake certainty.
When in doubt about location, use vision to localize the target instead of inventing coordinates from vibes.

## App-specific playbooks (recommended)

Build per-app routines for repetitive tasks instead of generic clicking.

### Launcher / search / start app playbook

Use this when the goal is "open app X" or "bring up tool Y".

1. check `GET /windows` first in case the app is already open
2. if present, use `POST /windows/action` to focus or restore it
3. if absent, prefer `POST /launch` when you know the executable path
4. if launch path is unknown but the OS launcher/search UI is available, use a keyboard-first flow:
   - open launcher (`win`, `cmd+space`, or app-specific shortcut depending on host)
   - type exact app name
   - wait for stable results with `POST /wait` or recapture
   - verify the result text with OCR or the `image` tool
   - press Enter or click the exact result once
5. verify the app window now exists or is focused

Do not keep relaunching if the window already exists; that’s sloppy.

### Dialog confirmation playbook

Use for modals like save/discard, delete confirmation, permission prompts, and installer dialogs.

1. capture the dialog region with `POST /zoom`
2. use OCR first for title/body/button labels
3. if button hierarchy or emphasis matters, inspect the zoomed screenshot with the `image` tool
4. identify the exact intended action (`Cancel`, `Save`, `Allow`, `Delete`, etc.)
5. for destructive actions, require explicit user confirmation unless already requested
6. click once and verify the dialog disappeared or changed state

Good verification targets:
- dialog title vanished
- expected next window appeared
- destructive side effect is visible and confirmed

### File picker playbook

Use for open/save dialogs.

1. verify the file picker window is focused
2. OCR the visible breadcrumb/path area, filename field, and button row
3. prefer keyboard-first entry when possible:
   - type or paste the target path/name into the focused field
   - use `tab` / `shift+tab` to move predictably between filename and action buttons
4. if the target path is uncertain, use OCR plus the `image` tool to identify the active field and selected folder/file row
5. verify the intended filename/path is visible before confirming
6. activate `Open` / `Save` once and verify the picker closes

If the picker stays open, stop and inspect why instead of hammering Enter like a maniac.

### Browser tab / window playbook

Use for browser navigation, tab targeting, or web app recovery.

1. use `GET /windows` to focus the correct browser window first
2. prefer keyboard-first navigation:
   - `ctrl+l` / `cmd+l` to focus the address bar
   - `ctrl+tab` / `ctrl+shift+tab` for tab movement when order is known
   - `ctrl+w` only for explicitly requested close actions
3. verify tab or page identity with OCR on the tab strip or page heading
4. if multiple similar tabs are open, zoom into the tab strip and use the `image` tool to distinguish active vs inactive tabs
5. after navigation, wait for visual stability or expected text before taking the next action
6. on shopping/checkouts, tighten crops around the buy box or checkout panel before reading button text; full-page OCR often misses the one thing that matters

Do not assume a page loaded just because the click landed. Verify it.

### Settings / preferences navigation playbook

Use when the task involves toggles, dropdowns, sidebars, or nested settings panels.

1. identify the current settings page with OCR on the heading/sidebar
2. use OCR to find the specific section label before trying to toggle anything
3. if the layout is dense, zoom into the relevant pane and use the `image` tool to distinguish labels from controls
4. prefer small reversible actions: one toggle, one dropdown, one field edit at a time
5. after each change, verify the control state changed visually or via visible text
6. if a save/apply button exists, treat it as a separate confirmation step and verify completion

Settings UIs love hiding side effects. Assume nothing.

### Dense app / control-strip playbook

Use for apps like OBS, IDEs, mixers, dashboards, or anything with tiny bottom-right control clusters.

1. focus the exact app window with `POST /windows/action`
2. capture the full target display once to confirm the window is actually frontmost
3. crop tightly around the suspected control strip with `POST /zoom`
4. run OCR on the crop, not the full screen
5. if labels are still ambiguous, ask the `image` tool a narrow question about the specific buttons
6. click once and immediately verify the control label changed (`Start Recording` -> `Stop Recording`, etc.)

Do not trust OCR taken from the wrong frontmost window. It will happily waste your time.

### Spotify playbook

- Focus app window before search/navigation.
- Prefer keyboard-first flow for song start:
  1) `Ctrl+L` (search)
  2) type exact query
  3) Enter
  4) verify exact song+artist text
  5) click/double-click row
  6) verify now-playing bar
- If now-playing does not match target track, stop and re-localize; do not keep clicking nearby rows.