196 lines
3.8 KiB
Markdown
196 lines
3.8 KiB
Markdown
# ScreenJob
|
|
|
|
ScreenJob is an autonomous desktop-and-terminal execution service.
|
|
It lets an LLM use controlled local tools (screen, click, type, shell) to complete GUI-heavy tasks on a real computer.
|
|
|
|
## What It Solves
|
|
|
|
- Runs agent-driven tasks that require a graphical interface.
|
|
- Exposes both CLI and HTTP API modes.
|
|
- Stores job history and events in SQLite.
|
|
- Streams live monitoring updates over WebSocket.
|
|
- Returns structured agent output as:
|
|
- `return`: human-readable completion message
|
|
- `data`: structured payload (for example command output)
|
|
|
|
## Core Features
|
|
|
|
- Tool-based agent loop (`execute_command`, `see_screen`, `enhance`, `click`, `type`, `press_key`, `sleep`, `task_complete`)
|
|
- Safety pre-check with override support
|
|
- Per-job tool disable list
|
|
- Live/final usage and cost estimates
|
|
- Read-only Tailwind monitoring UI
|
|
- Persistent job and event history
|
|
|
|
## Project Layout
|
|
|
|
```text
|
|
main.py
|
|
screenjob.py
|
|
requirements.txt
|
|
docker-compose.yml
|
|
src/
|
|
agent.py
|
|
app_main.py
|
|
cli.py
|
|
config.py
|
|
models.py
|
|
pricing.py
|
|
runtime.py
|
|
safety.py
|
|
server.py
|
|
storage.py
|
|
task_manager.py
|
|
ui.py
|
|
utils.py
|
|
tests/
|
|
test_agent_tools.py
|
|
test_pricing.py
|
|
test_server_api.py
|
|
test_storage.py
|
|
.gitea/workflows/ci.yml
|
|
```
|
|
|
|
## Setup
|
|
|
|
1. Install Python 3.11+.
|
|
2. Install dependencies:
|
|
|
|
```powershell
|
|
pip install -r requirements.txt
|
|
```
|
|
|
|
3. Create `.env` in project root:
|
|
|
|
```env
|
|
OPENAI_API_KEY=...
|
|
SCREENJOB_TOKEN=choose_a_strong_token
|
|
|
|
# Optional
|
|
SCREENJOB_DEFAULT_MODEL=gpt-5.4-mini
|
|
SCREENJOB_SAFETY_MODEL=gpt-5.4-mini
|
|
SCREENJOB_HOST=127.0.0.1
|
|
SCREENJOB_PORT=8787
|
|
DISABLE_UI=false
|
|
```
|
|
|
|
## Usage
|
|
|
|
### CLI
|
|
|
|
```powershell
|
|
python main.py run "Open amazon.de and go to my orders"
|
|
```
|
|
|
|
CLI JSON output includes both legacy and structured fields:
|
|
|
|
```json
|
|
{
|
|
"completed": true,
|
|
"result": "Task completed successfully",
|
|
"response": {
|
|
"return": "Task completed successfully",
|
|
"data": "file1.txt\nfile2.txt"
|
|
},
|
|
"return": "Task completed successfully",
|
|
"data": "file1.txt\nfile2.txt"
|
|
}
|
|
```
|
|
|
|
### Server
|
|
|
|
```powershell
|
|
python main.py server
|
|
```
|
|
|
|
Auth for all API routes:
|
|
|
|
- `Authorization: Bearer <SCREENJOB_TOKEN>`
|
|
- `X-ScreenJob-Token: <SCREENJOB_TOKEN>`
|
|
- Query fallback `?token=` (mainly for UI/websocket/artifact fetch)
|
|
|
|
### Create Job
|
|
|
|
`POST /api/jobs`
|
|
|
|
```json
|
|
{
|
|
"job": "run \"ls -a\" in C:/Users/username/Documents and return output",
|
|
"model": "gpt-5.4-mini",
|
|
"disabled_tools": [],
|
|
"safety_override": false
|
|
}
|
|
```
|
|
|
|
Response:
|
|
|
|
```json
|
|
{ "job_id": "job_..." }
|
|
```
|
|
|
|
### Job Status / History
|
|
|
|
- `GET /api/jobs/{job_id}`
|
|
- `GET /api/jobs/{job_id}/status`
|
|
- `GET /api/jobs/{job_id}/events`
|
|
- `GET /api/jobs`
|
|
- `POST /api/jobs/{job_id}/cancel`
|
|
- `GET /api/stats`
|
|
|
|
Each job payload includes:
|
|
|
|
- `result` (compat string)
|
|
- `response.return`
|
|
- `response.data`
|
|
- top-level `return` and `data` aliases
|
|
|
|
### Monitoring UI
|
|
|
|
- URL: `/`
|
|
- Read-only dashboard (no run controls)
|
|
- Requires token input
|
|
- Live updates via `/ws`
|
|
- Set `DISABLE_UI=true` to disable UI
|
|
|
|
## Agent Instructions (Practical)
|
|
|
|
- Prefer `execute_command` for deterministic actions (opening URLs, filesystem checks).
|
|
- Use `see_screen` before UI interaction.
|
|
- Use `enhance` when text is unclear.
|
|
- Use `press_key` for non-text keys (Enter, Tab, arrows, Escape).
|
|
- Use `click` offsets via `offset_up/down/left/right` and optional `sleep_after_seconds`.
|
|
- When done, call:
|
|
- `task_complete(return="...", data=...)`
|
|
|
|
`data` should contain useful structured output for the requester (text, object, list, etc.).
|
|
|
|
## Docker Compose
|
|
|
|
Run server in container:
|
|
|
|
```powershell
|
|
docker compose up --build
|
|
```
|
|
|
|
Service uses official Python image and reads `.env`.
|
|
|
|
## Verification
|
|
|
|
Local:
|
|
|
|
```powershell
|
|
pytest -q
|
|
```
|
|
|
|
CI:
|
|
|
|
- `.gitea/workflows/ci.yml` runs compile checks + tests on push/PR.
|
|
|
|
## Compatibility Entry Point
|
|
|
|
- `python screenjob.py "<job>"` remains supported as a wrapper to `main.py`.
|
|
|
|
## License
|
|
|
|
Apache License 2.0. See `LICENSE`.
|