# ScreenJob

ScreenJob is an autonomous desktop-and-terminal execution service.  
It lets an LLM use controlled local tools (screen, click, type, shell) to complete GUI-heavy tasks on a real computer.

## What It Solves

- Runs agent-driven tasks that require a graphical interface.
- Exposes both CLI and HTTP API modes.
- Stores job history and events in SQLite.
- Streams live monitoring updates over WebSocket.
- Returns structured agent output as:
  - `return`: human-readable completion message
  - `data`: structured payload (for example command output)
  - `verification`: final screen-capture metadata for completion accuracy checks

## Core Features

- Tool-based agent loop (`execute_command`, `see_screen`, `enhance`, `click`, `type`, `press_key`, `sleep`, `task_complete`)
- Safety pre-check with override support
- Per-job tool disable list
- Live/final usage and cost estimates
- Read-only Tailwind monitoring UI
- Persistent job and event history

## Project Layout

```text
main.py
screenjob.py
requirements.txt
start_backend.ps1
src/
  agent.py
  app_main.py
  cli.py
  config.py
  models.py
  pricing.py
  runtime.py
  safety.py
  server.py
  storage.py
  task_manager.py
  ui.py
  utils.py
tests/
  test_agent_tools.py
  test_pricing.py
  test_server_api.py
  test_storage.py
.gitea/workflows/ci.yml
```

## Setup

1. Install Python 3.11+.
2. Install dependencies:

```powershell
pip install -r requirements.txt
```

3. Create `.env` in project root:

```env
OPENAI_API_KEY=...
SCREENJOB_TOKEN=choose_a_strong_token

# Optional
SCREENJOB_DEFAULT_MODEL=gpt-5.4-mini
SCREENJOB_SAFETY_MODEL=gpt-5.4-mini
SCREENJOB_HOST=127.0.0.1
SCREENJOB_PORT=8787
DISABLE_UI=false
```

## Usage

### CLI

```powershell
python main.py run "Open amazon.de and go to my orders"
```

CLI JSON output includes both legacy and structured fields:

```json
{
  "completed": true,
  "result": "Task completed successfully",
  "response": {
    "return": "Task completed successfully",
    "data": "file1.txt\nfile2.txt"
  },
  "return": "Task completed successfully",
  "data": "file1.txt\nfile2.txt",
  "verification": {
    "ok": true,
    "path": "C:/.../screens/screen_final_verification_step_003.png"
  }
}
```

### Server

```powershell
python main.py server
```

Or use the PowerShell launcher:

```powershell
.\start_backend.ps1
```

Auth for all API routes:

- `Authorization: Bearer <SCREENJOB_TOKEN>`
- `X-ScreenJob-Token: <SCREENJOB_TOKEN>`
- Query fallback `?token=` (mainly for UI/websocket/artifact fetch)

### Create Job

`POST /api/jobs`

```json
{
  "job": "run \"ls -a\" in C:/Users/username/Documents and return output",
  "model": "gpt-5.4-mini",
  "disabled_tools": [],
  "safety_override": false
}
```

Response:

```json
{ "job_id": "job_..." }
```

### Job Status / History

- `GET /api/jobs/{job_id}`
- `GET /api/jobs/{job_id}/status`
- `GET /api/jobs/{job_id}/events`
- `GET /api/jobs`
- `POST /api/jobs/{job_id}/cancel`
- `GET /api/stats`

Each job payload includes:

- `result` (compat string)
- `response.return`
- `response.data`
- top-level `return` and `data` aliases
- `verification` (final screenshot path + metadata)

### Monitoring UI

- URL: `/`
- Read-only dashboard (no run controls)
- Requires token input
- Live updates via `/ws`
- Set `DISABLE_UI=true` to disable UI

## Agent Instructions (Practical)

- Prefer `execute_command` for deterministic actions (opening URLs, filesystem checks).
- Use `see_screen` before UI interaction.
- Use `enhance` when text is unclear.
- Use `press_key` for non-text keys (Enter, Tab, arrows, Escape).
- Use `click` offsets via `offset_up/down/left/right` and optional `sleep_after_seconds`.
- When done, call:
  - `task_complete(return="...", data=...)`
- A final verification screen capture is always taken automatically on completion.

`data` should contain useful structured output for the requester (text, object, list, etc.).

## Verification

Local:

```powershell
pytest -q
```

CI:

- `.gitea/workflows/ci.yml` runs compile checks + tests on push/PR.

## Compatibility Entry Point

- `python screenjob.py "<job>"` remains supported as a wrapper to `main.py`.

## License

Apache License 2.0. See `LICENSE`.