feat. Revision of Agent Lopp & major clean-up
CI / test (push) Successful in 8s

This commit is contained in:
2026-06-04 18:00:26 +02:00
parent 1f606475c5
commit 75a2600e4d
28 changed files with 753 additions and 1941 deletions
+23
View File
@@ -0,0 +1,23 @@
---
name: death-is-soon-wrap-up
description: Use when the user says "death is soon, wrap up" or wants the conversation closed out with a final note, concise summary, goodbye, and instruction to reset the chat.
---
# Death Is Soon Wrap Up
## Purpose
Use this skill to close a conversation cleanly and intentionally.
## Wrap-Up Flow
1. Note down anything that has a place to live.
1. If there are notes, tasks, summaries, or memory slots available, capture the important details there before ending.
1. Give the user one last short summary of the situation, decisions, and any next steps.
1. Say goodbye plainly.
1. Tell the user to reset the chat.
## Output Style
Keep the closing message short and calm.
Do not reopen the discussion after the goodbye unless the user asks for more.
+11 -53
View File
@@ -12,11 +12,12 @@ It lets an LLM use controlled local tools (screen, mouse, keyboard, clipboard, s
- Returns structured agent output as:
- `return`: human-readable completion message
- `data`: structured payload (for example command output)
- Cleans up old runs and history on startup (default retention: 7 days)
## Core Features
- Hybrid control model: screenshot grounding plus Windows-native window, dialog, and UI-element helpers when available
- Tool-based agent loop (`execute_command`, `see_screen`, `enhance`, `list_windows`, `find_window`, `focus_window`, `close_window`, `wait_for_window`, `wait_for_focus_change`, `detect_dialog`, `dialog_action`, `dialog_set_filename`, `wait_for_dialog_close`, `list_ui_elements`, `invoke_ui_element`, `set_ui_element_value`, `select_ui_element`, `wait_for_ui_element`, `click`, `scroll`, `drag`, `move_mouse`, `type`, `press_key`, `clipboard_get`, `clipboard_set`, `get_cursor_position`, `get_active_window`, `sleep`, `task_complete`)
- Tool-based agent loop (`execute_command`, `enhance`, `list_windows`, `find_window`, `focus_window`, `close_window`, `wait_for_window`, `wait_for_focus_change`, `detect_dialog`, `dialog_action`, `dialog_set_filename`, `wait_for_dialog_close`, `list_ui_elements`, `invoke_ui_element`, `set_ui_element_value`, `select_ui_element`, `wait_for_ui_element`, `click`, `scroll`, `drag`, `move_mouse`, `type`, `press_key`, `clipboard_get`, `clipboard_set`, `get_cursor_position`, `sleep`, `task_complete`)
- Safety pre-check with override support
- Per-job tool disable list
- Live/final usage and cost estimates
@@ -73,6 +74,7 @@ SCREENJOB_SAFETY_MODEL=gpt-5.4-mini
SCREENJOB_HOST=127.0.0.1
SCREENJOB_PORT=8787
DISABLE_UI=false
SCREENJOB_RETENTION_DAYS=7
```
## Usage
@@ -83,16 +85,11 @@ DISABLE_UI=false
python main.py run "Open amazon.de and go to my orders"
```
CLI JSON output includes both legacy and structured fields:
CLI JSON output:
```json
{
"completed": true,
"result": "Task completed successfully",
"response": {
"return": "Task completed successfully",
"data": "file1.txt\nfile2.txt"
},
"return": "Task completed successfully",
"data": "file1.txt\nfile2.txt"
}
@@ -147,43 +144,6 @@ If you need to start the backend manually, run:
.\start_backend.ps1
```
The legacy Windows service host remains in the tree for reference, but it is not the recommended path for GUI tasks.
### System Tray Icon (Windows)
Start tray icon now:
```powershell
powershell -NoProfile -ExecutionPolicy Bypass -STA -File .\screenjob_tray.ps1
```
Install startup shortcut (current user):
```powershell
.\install_tray_startup_shortcut.ps1
```
Install startup shortcut for all users:
```powershell
.\install_tray_startup_shortcut.ps1 -AllUsers
```
Remove startup shortcut:
```powershell
.\install_tray_startup_shortcut.ps1 -Remove
```
Tray menu actions:
- The service controls are for the legacy Windows service host.
- Refresh service status
- Start/Stop/Restart service (prompts for admin/UAC)
- Open dashboard URL from `.env` `SCREENJOB_HOST` / `SCREENJOB_PORT`
- Open service logs folder
- Exit tray icon process
Auth for all API routes:
- `Authorization: Bearer <SCREENJOB_TOKEN>`
@@ -225,10 +185,8 @@ Response:
Each job payload includes:
- `result` (compat string)
- `response.return`
- `response.data`
- top-level `return` and `data` aliases
- `return`
- `data`
### Monitoring UI
@@ -236,20 +194,20 @@ Each job payload includes:
- Read-only dashboard (no run controls)
- Requires token input
- Live updates via `/ws`
- Analytics dashboards for success rate by objective category and daily averages
- Analytics dashboard for success by objective category
- Set `DISABLE_UI=true` to disable UI
### Analytics API
- `GET /api/analytics`
- Returns objective-category success rates plus average steps/cost over time
- Returns objective-category success rates plus average steps/cost by category
## Agent Instructions (Practical)
- Prefer `execute_command` for deterministic actions (opening URLs, filesystem checks).
- First classify the current Windows surface, then choose the control channel.
- Prefer native window/dialog/element tools for focus changes, file pickers, modal confirmations, and browser-owned dialogs when available.
- Use `see_screen` before UI interaction.
- Use `enhance` for screen observation. Call it with no coordinate for a full-screen grid capture.
- Use `enhance` before clicking small/ambiguous targets; prefer `region="small"` for compact controls.
- Use `enhance` `mode="text"` for tiny labels/text, or `mode="ui"` for general UI.
- Optionally set `enhance` `scale` (2-6) for tighter zoom control.
@@ -261,11 +219,11 @@ Each job payload includes:
- Use `click` offsets via `offset_up/down/left/right`; set `button` and `click_count` there instead of inventing one-off click tools.
- Use `move_mouse` when you need hover-only behavior and `drag` for slider, selection, or window moves.
- Use `scroll` for vertical navigation; positive amounts scroll up and negative amounts scroll down.
- Use `clipboard_get` / `clipboard_set` for copy-paste workflows, `get_cursor_position` for cursor inspection, and `get_active_window` before interacting with uncertain focus.
- Use `clipboard_get` / `clipboard_set` for copy-paste workflows, `get_cursor_position` for cursor inspection, and rely on the injected foreground-window context before interacting with uncertain focus.
- If native automation is unavailable or disabled, ScreenJob falls back to screenshots plus mouse/keyboard control and emits fallback events.
- When done, call:
- `task_complete(return="...", data=...)`
- Before `task_complete`, verify expected on-screen content with `see_screen` (and `enhance` if needed), and include an `observed_result` summary in `data`.
- Before `task_complete`, verify expected on-screen content with the latest retained screen and `enhance` if needed, and include an `observed_result` summary in `data`.
Per-job `disabled_tools` must match the built-in tool allowlist. `task_complete` cannot be disabled.
-47
View File
@@ -1,47 +0,0 @@
[CmdletBinding(SupportsShouldProcess = $true)]
param(
[switch]$Remove,
[switch]$AllUsers
)
Set-StrictMode -Version Latest
$ErrorActionPreference = "Stop"
$scriptDir = Split-Path -Parent $PSCommandPath
$vbsLauncher = Join-Path $scriptDir "start_screenjob_tray_hidden.vbs"
$shortcutName = "ScreenJob Tray.lnk"
if (-not (Test-Path -LiteralPath $vbsLauncher)) {
throw "Launcher file not found: $vbsLauncher"
}
$startupFolder = if ($AllUsers) {
[Environment]::GetFolderPath("CommonStartup")
} else {
[Environment]::GetFolderPath("Startup")
}
$shortcutPath = Join-Path $startupFolder $shortcutName
if ($Remove) {
if (Test-Path -LiteralPath $shortcutPath) {
if ($PSCmdlet.ShouldProcess($shortcutPath, "Remove startup shortcut")) {
Remove-Item -LiteralPath $shortcutPath -Force
Write-Host "Removed startup shortcut: $shortcutPath"
}
} else {
Write-Host "No startup shortcut found at: $shortcutPath"
}
return
}
if ($PSCmdlet.ShouldProcess($shortcutPath, "Create startup shortcut")) {
$shell = New-Object -ComObject WScript.Shell
$shortcut = $shell.CreateShortcut($shortcutPath)
$shortcut.TargetPath = "$env:SystemRoot\System32\wscript.exe"
$shortcut.Arguments = '"' + $vbsLauncher + '"'
$shortcut.WorkingDirectory = $scriptDir
$shortcut.Description = "Launch ScreenJob tray icon at sign-in."
$shortcut.Save()
Write-Host "Created startup shortcut: $shortcutPath"
}
-307
View File
@@ -1,307 +0,0 @@
param(
[string]$ServiceName = "ScreenJobBackend"
)
Set-StrictMode -Version Latest
$ErrorActionPreference = "Stop"
Add-Type -AssemblyName System.Windows.Forms
Add-Type -AssemblyName System.Drawing
$scriptDir = Split-Path -Parent $MyInvocation.MyCommand.Path
$controlScript = Join-Path $scriptDir "tray_service_control.ps1"
$logsDir = Join-Path $scriptDir "screenjob_runs\service"
$defaultHost = "127.0.0.1"
$defaultPort = "8787"
function Read-EnvConfig {
param([string]$EnvFilePath)
$result = @{}
if (-not (Test-Path -LiteralPath $EnvFilePath)) {
return $result
}
foreach ($line in Get-Content -Path $EnvFilePath) {
$trimmed = $line.Trim()
if ($trimmed.Length -eq 0 -or $trimmed.StartsWith("#")) {
continue
}
$parts = $trimmed.Split("=", 2)
if ($parts.Count -eq 2) {
$key = $parts[0].Trim()
$value = $parts[1].Trim()
if (($value.StartsWith('"') -and $value.EndsWith('"')) -or ($value.StartsWith("'") -and $value.EndsWith("'"))) {
$value = $value.Substring(1, $value.Length - 2)
}
$result[$key] = $value
}
}
return $result
}
function Get-ServiceStatusSafe {
param([string]$Name)
try {
$svc = Get-Service -Name $Name -ErrorAction Stop
return $svc.Status.ToString()
} catch {
return "NotInstalled"
}
}
function Invoke-ServiceActionElevated {
param(
[Parameter(Mandatory = $true)][string]$Action,
[Parameter(Mandatory = $true)][string]$Name
)
if (-not (Test-Path -LiteralPath $controlScript)) {
[System.Windows.Forms.MessageBox]::Show(
"Missing control script: $controlScript",
"ScreenJob Tray",
[System.Windows.Forms.MessageBoxButtons]::OK,
[System.Windows.Forms.MessageBoxIcon]::Error
) | Out-Null
return
}
$argList = @(
"-NoProfile",
"-ExecutionPolicy", "Bypass",
"-File", "`"$controlScript`"",
"-Action", $Action,
"-ServiceName", $Name
)
try {
Start-Process -FilePath "powershell.exe" -ArgumentList $argList -Verb RunAs -WindowStyle Hidden | Out-Null
} catch {
# User canceled UAC prompt or launch failed.
}
}
function Get-DashboardUrl {
$envFile = Join-Path $scriptDir ".env"
$envVars = Read-EnvConfig -EnvFilePath $envFile
$dashboardHost = $defaultHost
$dashboardPort = $defaultPort
if ($envVars.ContainsKey("SCREENJOB_HOST") -and -not [string]::IsNullOrWhiteSpace($envVars["SCREENJOB_HOST"])) {
$dashboardHost = $envVars["SCREENJOB_HOST"]
}
if ($envVars.ContainsKey("SCREENJOB_PORT") -and -not [string]::IsNullOrWhiteSpace($envVars["SCREENJOB_PORT"])) {
$dashboardPort = $envVars["SCREENJOB_PORT"]
}
$connectHost = Resolve-ConnectHost -ConfiguredHost $dashboardHost
return "http://{0}:{1}/" -f $connectHost, $dashboardPort
}
function Resolve-ConnectHost {
param([string]$ConfiguredHost)
if ([string]::IsNullOrWhiteSpace($ConfiguredHost)) {
return "127.0.0.1"
}
switch ($ConfiguredHost.Trim().ToLowerInvariant()) {
"0.0.0.0" { return "127.0.0.1" }
"::" { return "127.0.0.1" }
"*" { return "127.0.0.1" }
default { return $ConfiguredHost }
}
}
function Get-HealthCheckHosts {
param([string]$ConfiguredHost)
if ([string]::IsNullOrWhiteSpace($ConfiguredHost)) {
return @("127.0.0.1", "localhost")
}
$normalized = $ConfiguredHost.Trim().ToLowerInvariant()
switch ($normalized) {
"0.0.0.0" { return @("127.0.0.1", "localhost", "::1") }
"::" { return @("127.0.0.1", "localhost", "::1") }
"*" { return @("127.0.0.1", "localhost", "::1") }
default { return @($ConfiguredHost) }
}
}
function Test-TcpEndpoint {
param(
[Parameter(Mandatory = $true)][string]$HostName,
[Parameter(Mandatory = $true)][int]$Port,
[int]$TimeoutMs = 1200
)
$client = New-Object System.Net.Sockets.TcpClient
try {
$async = $client.BeginConnect($HostName, $Port, $null, $null)
$connected = $async.AsyncWaitHandle.WaitOne($TimeoutMs, $false)
if (-not $connected) {
return $false
}
$client.EndConnect($async) | Out-Null
return $true
} catch {
return $false
} finally {
$client.Dispose()
}
}
function Get-BackendReachability {
$envFile = Join-Path $scriptDir ".env"
$envVars = Read-EnvConfig -EnvFilePath $envFile
$configuredHost = $defaultHost
$configuredPort = $defaultPort
if ($envVars.ContainsKey("SCREENJOB_HOST") -and -not [string]::IsNullOrWhiteSpace($envVars["SCREENJOB_HOST"])) {
$configuredHost = $envVars["SCREENJOB_HOST"]
}
if ($envVars.ContainsKey("SCREENJOB_PORT") -and -not [string]::IsNullOrWhiteSpace($envVars["SCREENJOB_PORT"])) {
$configuredPort = $envVars["SCREENJOB_PORT"]
}
$portNumber = 8787
[void][int]::TryParse([string]$configuredPort, [ref]$portNumber)
$hostsToTry = Get-HealthCheckHosts -ConfiguredHost $configuredHost
foreach ($candidateHost in $hostsToTry) {
if (Test-TcpEndpoint -HostName $candidateHost -Port $portNumber) {
return $true
}
}
return $false
}
function Update-TrayState {
param(
[System.Windows.Forms.NotifyIcon]$NotifyIcon,
[System.Windows.Forms.ToolStripMenuItem]$StatusItem,
[string]$Name
)
$status = Get-ServiceStatusSafe -Name $Name
$isBackendReachable = Get-BackendReachability
$displayStatus = $status
if ($status -eq "Running" -and -not $isBackendReachable) {
$displayStatus = "Running (Backend Down)"
} elseif ($status -eq "Stopped" -and $isBackendReachable) {
$displayStatus = "Stopped (Backend Up)"
} elseif ($status -eq "NotInstalled" -and $isBackendReachable) {
$displayStatus = "NotInstalled (Backend Up)"
}
$StatusItem.Text = "Status: $displayStatus"
switch ($displayStatus) {
"Running" {
$NotifyIcon.Icon = [System.Drawing.SystemIcons]::Information
}
"Stopped" {
$NotifyIcon.Icon = [System.Drawing.SystemIcons]::Warning
}
default {
$NotifyIcon.Icon = [System.Drawing.SystemIcons]::Error
}
}
$tooltip = "ScreenJob Backend: $displayStatus"
if ($tooltip.Length -gt 63) {
$tooltip = $tooltip.Substring(0, 63)
}
$NotifyIcon.Text = $tooltip
}
$appContext = New-Object System.Windows.Forms.ApplicationContext
$notifyIcon = New-Object System.Windows.Forms.NotifyIcon
$notifyIcon.Visible = $false
$menu = New-Object System.Windows.Forms.ContextMenuStrip
$statusItem = New-Object System.Windows.Forms.ToolStripMenuItem "Status: Unknown"
$statusItem.Enabled = $false
$refreshItem = New-Object System.Windows.Forms.ToolStripMenuItem "Refresh Status"
$refreshItem.Add_Click({
Update-TrayState -NotifyIcon $notifyIcon -StatusItem $statusItem -Name $ServiceName
})
$startItem = New-Object System.Windows.Forms.ToolStripMenuItem "Start Service (Admin)"
$startItem.Add_Click({
Invoke-ServiceActionElevated -Action "start" -Name $ServiceName
})
$stopItem = New-Object System.Windows.Forms.ToolStripMenuItem "Stop Service (Admin)"
$stopItem.Add_Click({
Invoke-ServiceActionElevated -Action "stop" -Name $ServiceName
})
$restartItem = New-Object System.Windows.Forms.ToolStripMenuItem "Restart Service (Admin)"
$restartItem.Add_Click({
Invoke-ServiceActionElevated -Action "restart" -Name $ServiceName
})
$dashboardItem = New-Object System.Windows.Forms.ToolStripMenuItem "Open Dashboard"
$dashboardItem.Add_Click({
$url = Get-DashboardUrl
Start-Process $url | Out-Null
})
$logsItem = New-Object System.Windows.Forms.ToolStripMenuItem "Open Service Logs"
$logsItem.Add_Click({
if (-not (Test-Path -LiteralPath $logsDir)) {
New-Item -ItemType Directory -Path $logsDir -Force | Out-Null
}
Start-Process explorer.exe $logsDir | Out-Null
})
$openFolderItem = New-Object System.Windows.Forms.ToolStripMenuItem "Open Project Folder"
$openFolderItem.Add_Click({
Start-Process explorer.exe $scriptDir | Out-Null
})
$exitItem = New-Object System.Windows.Forms.ToolStripMenuItem "Exit Tray"
$exitItem.Add_Click({
$refreshTimer.Stop()
$notifyIcon.Visible = $false
$notifyIcon.Dispose()
$menu.Dispose()
$appContext.ExitThread()
})
[void]$menu.Items.Add($statusItem)
[void]$menu.Items.Add($refreshItem)
[void]$menu.Items.Add((New-Object System.Windows.Forms.ToolStripSeparator))
[void]$menu.Items.Add($startItem)
[void]$menu.Items.Add($stopItem)
[void]$menu.Items.Add($restartItem)
[void]$menu.Items.Add((New-Object System.Windows.Forms.ToolStripSeparator))
[void]$menu.Items.Add($dashboardItem)
[void]$menu.Items.Add($logsItem)
[void]$menu.Items.Add($openFolderItem)
[void]$menu.Items.Add((New-Object System.Windows.Forms.ToolStripSeparator))
[void]$menu.Items.Add($exitItem)
$notifyIcon.ContextMenuStrip = $menu
$notifyIcon.Visible = $true
$notifyIcon.Add_DoubleClick({
$url = Get-DashboardUrl
Start-Process $url | Out-Null
})
$refreshTimer = New-Object System.Windows.Forms.Timer
$refreshTimer.Interval = 5000
$refreshTimer.Add_Tick({
Update-TrayState -NotifyIcon $notifyIcon -StatusItem $statusItem -Name $ServiceName
})
Update-TrayState -NotifyIcon $notifyIcon -StatusItem $statusItem -Name $ServiceName
$refreshTimer.Start()
[System.Windows.Forms.Application]::Run($appContext)
@@ -1,138 +0,0 @@
using System.Diagnostics;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;
namespace ScreenJob.WindowsServiceHost;
internal sealed class BackendProcessService : BackgroundService
{
private readonly ILogger<BackendProcessService> _logger;
private readonly ServiceOptions _options;
private readonly object _logLock = new();
private Process? _backendProcess;
private string _stdoutLogPath = string.Empty;
private string _stderrLogPath = string.Empty;
public BackendProcessService(ILogger<BackendProcessService> logger, ServiceOptions options)
{
_logger = logger;
_options = options;
}
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
Directory.CreateDirectory(_options.LogDirectory);
_stdoutLogPath = Path.Combine(_options.LogDirectory, "backend-service.stdout.log");
_stderrLogPath = Path.Combine(_options.LogDirectory, "backend-service.stderr.log");
LogStdOut("Service host starting backend process.");
LogStdOut($"Script: {_options.BackendScriptPath}");
LogStdOut($"Working directory: {_options.WorkingDirectory}");
var powershellPath = Path.Combine(
Environment.GetFolderPath(Environment.SpecialFolder.Windows),
"System32",
"WindowsPowerShell",
"v1.0",
"powershell.exe");
var startInfo = new ProcessStartInfo
{
FileName = powershellPath,
Arguments = $"-NoProfile -ExecutionPolicy Bypass -File \"{_options.BackendScriptPath}\"",
WorkingDirectory = _options.WorkingDirectory,
RedirectStandardOutput = true,
RedirectStandardError = true,
UseShellExecute = false,
CreateNoWindow = true
};
_backendProcess = new Process { StartInfo = startInfo };
if (!_backendProcess.Start())
{
throw new InvalidOperationException("Failed to start backend process.");
}
LogStdOut($"Backend process started with PID {_backendProcess.Id}.");
_logger.LogInformation("Backend process started with PID {Pid}.", _backendProcess.Id);
var stdoutPump = PumpStreamAsync(_backendProcess.StandardOutput, LogStdOut, stoppingToken);
var stderrPump = PumpStreamAsync(_backendProcess.StandardError, LogStdErr, stoppingToken);
try
{
await _backendProcess.WaitForExitAsync(stoppingToken);
var exitCode = _backendProcess.ExitCode;
LogStdErr($"Backend process exited unexpectedly with code {exitCode}.");
_logger.LogError("Backend process exited unexpectedly with code {ExitCode}.", exitCode);
Environment.ExitCode = exitCode == 0 ? 1 : exitCode;
throw new InvalidOperationException(
$"Backend process ended unexpectedly. Service host exit code: {Environment.ExitCode}.");
}
catch (OperationCanceledException)
{
LogStdOut("Service stop requested.");
}
finally
{
await Task.WhenAll(stdoutPump, stderrPump);
}
}
public override async Task StopAsync(CancellationToken cancellationToken)
{
if (_backendProcess is { HasExited: false })
{
try
{
LogStdOut("Stopping backend process.");
_backendProcess.Kill(entireProcessTree: true);
}
catch (Exception ex)
{
LogStdErr($"Failed to stop backend process cleanly: {ex.Message}");
_logger.LogError(ex, "Failed to stop backend process cleanly.");
}
}
await base.StopAsync(cancellationToken);
}
private async Task PumpStreamAsync(
StreamReader reader,
Action<string> sink,
CancellationToken stoppingToken)
{
while (!stoppingToken.IsCancellationRequested)
{
var line = await reader.ReadLineAsync();
if (line is null)
{
break;
}
sink(line);
}
}
private void LogStdOut(string message)
{
WriteLog(_stdoutLogPath, message);
}
private void LogStdErr(string message)
{
WriteLog(_stderrLogPath, message);
}
private void WriteLog(string path, string message)
{
var stamp = DateTimeOffset.Now.ToString("yyyy-MM-dd HH:mm:ss");
var line = $"[{stamp}] {message}{Environment.NewLine}";
lock (_logLock)
{
File.AppendAllText(path, line);
}
}
}
@@ -1,18 +0,0 @@
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using ScreenJob.WindowsServiceHost;
var options = ServiceOptions.Parse(args);
Host.CreateDefaultBuilder(args)
.UseWindowsService(serviceOptions =>
{
serviceOptions.ServiceName = "ScreenJobBackend";
})
.ConfigureServices(services =>
{
services.AddSingleton(options);
services.AddHostedService<BackendProcessService>();
})
.Build()
.Run();
@@ -1,12 +0,0 @@
<Project Sdk="Microsoft.NET.Sdk.Worker">
<PropertyGroup>
<TargetFramework>net10.0-windows</TargetFramework>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
<OutputType>Exe</OutputType>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="Microsoft.Extensions.Hosting.WindowsServices" Version="10.0.0" />
</ItemGroup>
</Project>
@@ -1,77 +0,0 @@
namespace ScreenJob.WindowsServiceHost;
internal sealed record ServiceOptions(
string BackendScriptPath,
string WorkingDirectory,
string LogDirectory)
{
public static ServiceOptions Parse(string[] args)
{
var map = new Dictionary<string, string>(StringComparer.OrdinalIgnoreCase);
for (var i = 0; i < args.Length; i++)
{
var raw = args[i];
if (!raw.StartsWith("--", StringComparison.Ordinal))
{
continue;
}
var key = raw[2..];
if (string.IsNullOrWhiteSpace(key))
{
continue;
}
if (i + 1 < args.Length && !args[i + 1].StartsWith("--", StringComparison.Ordinal))
{
map[key] = args[++i];
}
else
{
map[key] = "true";
}
}
if (!map.TryGetValue("backend-script", out var backendScript) || string.IsNullOrWhiteSpace(backendScript))
{
throw new ArgumentException("Missing required argument: --backend-script <absolute-path-to-start_backend.ps1>.");
}
if (!Path.IsPathRooted(backendScript))
{
throw new ArgumentException("The --backend-script value must be an absolute path.");
}
if (!File.Exists(backendScript))
{
throw new FileNotFoundException("Backend script not found.", backendScript);
}
if (!map.TryGetValue("working-dir", out var workingDir) || string.IsNullOrWhiteSpace(workingDir))
{
workingDir = Path.GetDirectoryName(backendScript)
?? throw new ArgumentException("Could not resolve working directory from backend script path.");
}
if (!Path.IsPathRooted(workingDir))
{
throw new ArgumentException("The --working-dir value must be an absolute path.");
}
if (!map.TryGetValue("log-dir", out var logDir) || string.IsNullOrWhiteSpace(logDir))
{
logDir = Path.Combine(workingDir, "screenjob_runs", "service");
}
if (!Path.IsPathRooted(logDir))
{
throw new ArgumentException("The --log-dir value must be an absolute path.");
}
return new ServiceOptions(
Path.GetFullPath(backendScript),
Path.GetFullPath(workingDir),
Path.GetFullPath(logDir));
}
}
+317 -239
View File
File diff suppressed because it is too large Load Diff
+1
View File
@@ -31,6 +31,7 @@ def main(argv: list[str] | None = None) -> int:
print(" SCREENJOB_TOKEN=...")
print(" DISABLE_UI=true|false (optional)")
print(" SCREENJOB_PROHIBITED_KEY_COMBOS=ctrl+shift+s,alt+f4 (optional)")
print(" SCREENJOB_RETENTION_DAYS=7 (optional)")
return 0
server.main()
return 0
+63 -39
View File
@@ -2,15 +2,18 @@ from __future__ import annotations
import argparse
import json
import signal
import sys
import threading
from pathlib import Path
from .agent import normalize_disabled_tools
from .config import load_app_config
from .desktop_overlay import get_desktop_overlay_manager
from .models import RuntimeOptions
from .runtime import create_openai_client, run_job
from .safety import assess_task_safety
from .storage import HistoryDB
from .utils import cleanup_old_run_artifacts
def build_parser() -> argparse.ArgumentParser:
@@ -45,12 +48,12 @@ def build_parser() -> argparse.ArgumentParser:
parser.add_argument(
"--max-visual-context-images",
type=int,
default=3,
help="Maximum screenshots/enhanced images retained in model-visible context during rebases.",
default=None,
help="Maximum recent screen images retained in model-visible context during rebases. Defaults to SCREENJOB_MAX_VISUAL_CONTEXT_IMAGES or 3.",
)
parser.add_argument(
"--native-automation-mode",
choices=["off", "prefer", "require_fallback"],
choices=["off", "prefer"],
default="prefer",
help="How strongly the agent should prefer Windows-native automation helpers over pixel fallback.",
)
@@ -80,7 +83,8 @@ def build_parser() -> argparse.ArgumentParser:
)
parser.add_argument(
"--pretty-logs",
action="store_true",
action=argparse.BooleanOptionalAction,
default=False,
help="Emit expanded multi-line tool call/result logs for easier debugging.",
)
parser.add_argument("--disable-tool", action="append", default=[], help="Disable a tool by name.")
@@ -105,6 +109,9 @@ def main(argv: list[str] | None = None) -> int:
print("ERROR: Missing OPENAI_API_KEY in environment/.env", file=sys.stderr)
return 2
HistoryDB(config.db_path).prune_older_than(config.retention_days)
cleanup_old_run_artifacts(config.runs_dir, config.retention_days)
model = args.model or config.default_model
try:
disabled_tools = normalize_disabled_tools(args.disable_tool)
@@ -124,8 +131,6 @@ def main(argv: list[str] | None = None) -> int:
json.dumps(
{
"completed": False,
"result": f"Blocked by safety check: {reason}",
"response": {"return": f"Blocked by safety check: {reason}", "data": parsed},
"return": f"Blocked by safety check: {reason}",
"data": parsed,
"safety": parsed,
@@ -144,7 +149,14 @@ def main(argv: list[str] | None = None) -> int:
click_pause=args.click_pause,
reasoning_effort=args.reasoning_effort,
screen_context_decay_steps=max(0, int(args.screen_context_decay_steps)),
max_visual_context_images=max(0, int(args.max_visual_context_images)),
max_visual_context_images=max(
0,
int(
config.max_visual_context_images_default
if args.max_visual_context_images is None
else args.max_visual_context_images
),
),
native_automation_mode=args.native_automation_mode,
dialog_timeout_seconds=max(0.5, float(args.dialog_timeout_seconds)),
focus_timeout_seconds=max(0.5, float(args.focus_timeout_seconds)),
@@ -154,41 +166,53 @@ def main(argv: list[str] | None = None) -> int:
disable_tools=set(disabled_tools),
prohibited_key_combos=set(config.prohibited_key_combos),
)
try:
result, artifacts = run_job(
api_key=config.openai_api_key,
objective=args.job,
options=options,
runs_base=config.runs_dir,
no_failsafe=args.no_failsafe,
)
except KeyboardInterrupt:
print(json.dumps({"completed": False, "result": "Interrupted by user."}, ensure_ascii=False, indent=2))
return 130
except Exception as exc: # noqa: BLE001
print(
json.dumps(
{"completed": False, "result": f"Fatal error: {type(exc).__name__}: {exc}"},
ensure_ascii=False,
indent=2,
),
file=sys.stderr,
)
return 1
cancel_event = threading.Event()
interrupt_state = {"count": 0}
previous_sigint_handler = signal.getsignal(signal.SIGINT)
if result.completed:
get_desktop_overlay_manager().show_completion(
job_id=artifacts.run_id,
objective=args.job,
return_message=result.return_message,
steps=result.steps,
elapsed_seconds=max(0.0, float(result.ended_at - result.started_at)),
)
def _handle_sigint(_signum, _frame):
interrupt_state["count"] += 1
cancel_event.set()
if interrupt_state["count"] == 1:
print("\nInterrupt received, cancelling run... Press Ctrl+C again to abort immediately.", file=sys.stderr)
return
raise KeyboardInterrupt
signal.signal(signal.SIGINT, _handle_sigint)
try:
try:
result, artifacts = run_job(
api_key=config.openai_api_key,
objective=args.job,
options=options,
runs_base=config.runs_dir,
no_failsafe=args.no_failsafe,
cancel_event=cancel_event,
)
except KeyboardInterrupt:
print(json.dumps({"completed": False, "return": "Interrupted by user.", "data": None}, ensure_ascii=False, indent=2))
return 130
except Exception as exc: # noqa: BLE001
print(
json.dumps(
{"completed": False, "return": f"Fatal error: {type(exc).__name__}: {exc}", "data": None},
ensure_ascii=False,
indent=2,
),
file=sys.stderr,
)
return 1
finally:
signal.signal(signal.SIGINT, previous_sigint_handler)
if result.cancelled:
usage = result.usage.to_dict()
estimated_cost = usage.get("estimated_cost_usd")
if estimated_cost is not None:
print(f"Estimated cost before cancel: ${float(estimated_cost):.6f}", file=sys.stderr)
payload = {
"completed": result.completed,
"result": result.return_message,
"response": {"return": result.return_message, "data": result.data},
"return": result.return_message,
"data": result.data,
"steps": result.steps,
+6
View File
@@ -32,7 +32,9 @@ class AppConfig:
port: int
runs_dir: Path
db_path: Path
max_visual_context_images_default: int = 3
prohibited_key_combos: tuple[str, ...] = ()
retention_days: int = 7
def load_app_config(cwd: Path) -> AppConfig:
@@ -46,7 +48,9 @@ def load_app_config(cwd: Path) -> AppConfig:
runs_dir = cwd / "screenjob_runs"
db_path = cwd / "screenjob.db"
disable_ui = _env_bool("DISABLE_UI", default=False)
max_visual_context_images_default = max(0, int(os.getenv("SCREENJOB_MAX_VISUAL_CONTEXT_IMAGES", "3").strip() or "3"))
prohibited_key_combos = tuple(_env_csv("SCREENJOB_PROHIBITED_KEY_COMBOS"))
retention_days = max(1, int(os.getenv("SCREENJOB_RETENTION_DAYS", "7").strip() or "7"))
return AppConfig(
openai_api_key=openai_api_key,
screenjob_token=screenjob_token,
@@ -57,5 +61,7 @@ def load_app_config(cwd: Path) -> AppConfig:
port=port,
runs_dir=runs_dir,
db_path=db_path,
max_visual_context_images_default=max_visual_context_images_default,
prohibited_key_combos=prohibited_key_combos,
retention_days=retention_days,
)
-286
View File
@@ -1,286 +0,0 @@
from __future__ import annotations
import logging
import os
import queue
import threading
from dataclasses import dataclass
from typing import Any
try:
import winsound
except Exception: # noqa: BLE001
winsound = None
@dataclass(frozen=True)
class CompletionOverlayPayload:
job_id: str
objective: str
return_message: str
steps: int
elapsed_seconds: float
class DesktopOverlayManager:
def __init__(self, logger: logging.Logger | None = None, *, auto_dismiss_seconds: float = 10.0) -> None:
self.logger = logger or logging.getLogger("screenjob.overlay")
self._queue: queue.Queue[CompletionOverlayPayload] = queue.Queue()
self._thread: threading.Thread | None = None
self._lock = threading.Lock()
self._ready = threading.Event()
self._disabled = False
self._warned = False
self._auto_dismiss_ms = max(0, int(round(float(auto_dismiss_seconds) * 1000)))
def _play_completion_sound(self) -> None:
if os.name != "nt" or winsound is None:
return
try:
winsound.MessageBeep(winsound.MB_ICONASTERISK)
except Exception as exc: # noqa: BLE001
self.logger.debug("Completion sound failed (%s: %s)", type(exc).__name__, exc)
def show_completion(
self,
*,
job_id: str,
objective: str,
return_message: str,
steps: int,
elapsed_seconds: float,
) -> None:
self._play_completion_sound()
if os.name != "nt":
self._disable_once("Desktop completion HUD is only enabled on Windows.")
return
if not self._ensure_thread():
return
self._queue.put(
CompletionOverlayPayload(
job_id=job_id,
objective=objective,
return_message=return_message,
steps=max(0, int(steps)),
elapsed_seconds=max(0.0, float(elapsed_seconds)),
)
)
def _ensure_thread(self) -> bool:
with self._lock:
if self._disabled:
return False
if self._thread is None or not self._thread.is_alive():
self._ready.clear()
self._thread = threading.Thread(target=self._ui_main, name="screenjob-overlay", daemon=True)
self._thread.start()
self._ready.wait(timeout=2.0)
return not self._disabled
def _disable_once(self, reason: str) -> None:
with self._lock:
self._disabled = True
already_warned = self._warned
self._warned = True
self._ready.set()
if not already_warned:
self.logger.warning("%s Overlay notifications disabled.", reason)
def _format_elapsed(self, elapsed_seconds: float) -> str:
total_seconds = max(0, int(round(elapsed_seconds)))
minutes, seconds = divmod(total_seconds, 60)
hours, minutes = divmod(minutes, 60)
if hours:
return f"{hours}h {minutes}m {seconds}s"
if minutes:
return f"{minutes}m {seconds}s"
return f"{seconds}s"
def _shorten(self, text: str, limit: int) -> str:
raw = " ".join(str(text or "").split())
if len(raw) <= limit:
return raw
return raw[: max(0, limit - 1)].rstrip() + "..."
def _ui_main(self) -> None:
try:
import tkinter as tk
except Exception as exc: # noqa: BLE001
self._disable_once(f"tkinter is unavailable ({type(exc).__name__}: {exc}).")
return
try:
root = tk.Tk()
root.withdraw()
root.update_idletasks()
except Exception as exc: # noqa: BLE001
self._disable_once(f"Desktop overlay could not initialize ({type(exc).__name__}: {exc}).")
return
cards: list[dict[str, Any]] = []
self._ready.set()
def reposition() -> None:
screen_width = root.winfo_screenwidth()
top = 24
for entry in cards:
window = entry["window"]
if not bool(window.winfo_exists()):
continue
window.update_idletasks()
width = max(320, int(window.winfo_width() or 360))
height = max(120, int(window.winfo_height() or 160))
left = max(12, screen_width - width - 24)
window.geometry(f"{width}x{height}+{left}+{top}")
top += height + 16
def dismiss(window: Any) -> None:
for index, entry in enumerate(list(cards)):
if entry["window"] is window:
after_id = entry.get("after_id")
if after_id is not None:
try:
window.after_cancel(after_id)
except Exception: # noqa: BLE001
pass
cards.pop(index)
break
try:
if bool(window.winfo_exists()):
window.destroy()
except Exception: # noqa: BLE001
pass
if cards:
reposition()
def add_card(payload: CompletionOverlayPayload) -> None:
card = tk.Toplevel(root)
card.withdraw()
card.overrideredirect(True)
card.attributes("-topmost", True)
card.configure(bg="#0f172a")
frame = tk.Frame(card, bg="#0f172a", highlightthickness=1, highlightbackground="#22c55e", bd=0)
frame.pack(fill="both", expand=True)
close_button = tk.Button(
frame,
text="×",
command=lambda win=card: dismiss(win),
bg="#0f172a",
fg="#cbd5e1",
activebackground="#111827",
activeforeground="#ffffff",
relief="flat",
borderwidth=0,
font=("Segoe UI", 14, "bold"),
padx=6,
pady=0,
)
close_button.place(relx=1.0, x=-8, y=6, anchor="ne")
header = tk.Label(
frame,
text="Completed",
bg="#0f172a",
fg="#86efac",
font=("Segoe UI", 10, "bold"),
anchor="w",
)
header.pack(fill="x", padx=14, pady=(12, 2))
title = tk.Label(
frame,
text=self._shorten(payload.objective, 72) or "Job complete",
bg="#0f172a",
fg="#f8fafc",
font=("Segoe UI", 11, "bold"),
justify="left",
wraplength=320,
anchor="w",
)
title.pack(fill="x", padx=14)
job_row = tk.Label(
frame,
text=f"Job {payload.job_id}",
bg="#0f172a",
fg="#94a3b8",
font=("Segoe UI", 9),
justify="left",
anchor="w",
)
job_row.pack(fill="x", padx=14, pady=(2, 8))
message = tk.Label(
frame,
text=self._shorten(payload.return_message, 180) or "Task completed.",
bg="#0f172a",
fg="#e2e8f0",
font=("Segoe UI", 9),
justify="left",
wraplength=320,
anchor="w",
)
message.pack(fill="x", padx=14)
footer = tk.Label(
frame,
text=f"{payload.steps} step(s) | {self._format_elapsed(payload.elapsed_seconds)}",
bg="#0f172a",
fg="#94a3b8",
font=("Segoe UI", 9),
justify="left",
anchor="w",
)
footer.pack(fill="x", padx=14, pady=(10, 12))
after_id = None
if self._auto_dismiss_ms > 0:
after_id = card.after(self._auto_dismiss_ms, lambda win=card: dismiss(win))
cards.insert(0, {"window": card, "after_id": after_id})
while len(cards) > 3:
stale = cards.pop()
try:
stale_after_id = stale.get("after_id")
if stale_after_id is not None:
stale["window"].after_cancel(stale_after_id)
stale["window"].destroy()
except Exception: # noqa: BLE001
pass
card.update_idletasks()
reposition()
card.deiconify()
def pump_queue() -> None:
try:
while True:
add_card(self._queue.get_nowait())
except queue.Empty:
pass
try:
root.after(120, pump_queue)
except Exception: # noqa: BLE001
self._disable_once("Desktop overlay event loop stopped unexpectedly.")
pump_queue()
try:
root.mainloop()
except Exception as exc: # noqa: BLE001
self._disable_once(f"Desktop overlay main loop failed ({type(exc).__name__}: {exc}).")
_overlay_singleton: DesktopOverlayManager | None = None
_overlay_lock = threading.Lock()
def get_desktop_overlay_manager(logger: logging.Logger | None = None) -> DesktopOverlayManager:
global _overlay_singleton
with _overlay_lock:
if _overlay_singleton is None:
_overlay_singleton = DesktopOverlayManager(logger=logger)
elif logger is not None:
_overlay_singleton.logger = logger
return _overlay_singleton
+24 -16
View File
@@ -20,6 +20,9 @@ else:
_PYAUTOGUI_IMPORT_ERROR = None
_DESKTOP_RUN_LOCK = threading.Lock()
def create_openai_client(api_key: str) -> OpenAI:
return OpenAI(api_key=api_key)
@@ -44,20 +47,25 @@ def run_job(
pyautogui.FAILSAFE = not no_failsafe
pyautogui.PAUSE = 0.05
artifacts = setup_artifacts(runs_base)
active_logger = logger or setup_logger(artifacts.log_file, verbose=True)
active_logger.info("ScreenJob booting. Artifacts: %s", str(artifacts.root_dir.resolve()))
active_logger.info("PyAutoGUI FAILSAFE=%s", pyautogui.FAILSAFE)
if not _DESKTOP_RUN_LOCK.acquire(blocking=False):
raise RuntimeError("Another ScreenJob run is already active on this desktop.")
try:
artifacts = setup_artifacts(runs_base)
active_logger = logger or setup_logger(artifacts.log_file, verbose=True)
active_logger.info("ScreenJob booting. Artifacts: %s", str(artifacts.root_dir.resolve()))
active_logger.info("PyAutoGUI FAILSAFE=%s", pyautogui.FAILSAFE)
client = create_openai_client(api_key)
agent = ScreenJobAgent(
client=client,
logger=active_logger,
artifacts=artifacts,
options=options,
cancel_event=cancel_event,
event_callback=event_callback,
)
result = agent.run(objective)
active_logger.info("Run finished. completed=%s elapsed=%.2fs", result.completed, result.ended_at - result.started_at)
return result, artifacts
client = create_openai_client(api_key)
agent = ScreenJobAgent(
client=client,
logger=active_logger,
artifacts=artifacts,
options=options,
cancel_event=cancel_event,
event_callback=event_callback,
)
result = agent.run(objective)
active_logger.info("Run finished. completed=%s elapsed=%.2fs", result.completed, result.ended_at - result.started_at)
return result, artifacts
finally:
_DESKTOP_RUN_LOCK.release()
+11 -5
View File
@@ -17,7 +17,7 @@ from .config import AppConfig, load_app_config
from .storage import HistoryDB
from .task_manager import JobManager
from .ui import monitoring_js_path, monitoring_page_html
from .utils import utc_now_iso
from .utils import cleanup_old_run_artifacts, utc_now_iso
class CreateJobRequest(BaseModel):
@@ -29,8 +29,8 @@ class CreateJobRequest(BaseModel):
click_pause: float = Field(0.10, ge=0.0, le=2.0)
reasoning_effort: str = Field("medium", pattern="^(low|medium|high)$")
screen_context_decay_steps: int = Field(4, ge=0, le=50)
max_visual_context_images: int = Field(3, ge=0, le=12)
native_automation_mode: str = Field("prefer", pattern="^(off|prefer|require_fallback)$")
max_visual_context_images: int | None = Field(default=None, ge=0, le=12)
native_automation_mode: str = Field("prefer", pattern="^(off|prefer)$")
dialog_timeout_seconds: float = Field(12.0, ge=0.5, le=120.0)
focus_timeout_seconds: float = Field(8.0, ge=0.5, le=120.0)
ui_element_timeout_seconds: float = Field(8.0, ge=0.5, le=120.0)
@@ -183,7 +183,7 @@ def _build_replay_payload(job_id: str, job: dict[str, Any], events: list[dict[st
width = _safe_int(size.get("width"))
height = _safe_int(size.get("height"))
is_fullscreen = (
str(payload.get("kind") or "") == "see_screen"
(bool(image_meta.get("full_screen")) or str(payload.get("kind") or "") == "see_screen")
and bool(image_meta.get("grid"))
and isinstance(width, int)
and isinstance(height, int)
@@ -262,6 +262,8 @@ def create_app(config: AppConfig | None = None) -> FastAPI:
raise RuntimeError("SCREENJOB_TOKEN is required in environment or .env.")
db = HistoryDB(app_config.db_path)
db.prune_older_than(app_config.retention_days)
cleanup_old_run_artifacts(app_config.runs_dir, app_config.retention_days)
ws_hub = _WebSocketHub()
manager = JobManager(config=app_config, db=db, broadcast=ws_hub.broadcast_from_thread)
@@ -316,7 +318,11 @@ def create_app(config: AppConfig | None = None) -> FastAPI:
click_pause=payload.click_pause,
reasoning_effort=payload.reasoning_effort,
screen_context_decay_steps=payload.screen_context_decay_steps,
max_visual_context_images=payload.max_visual_context_images,
max_visual_context_images=(
app_config.max_visual_context_images_default
if payload.max_visual_context_images is None
else payload.max_visual_context_images
),
native_automation_mode=payload.native_automation_mode,
dialog_timeout_seconds=payload.dialog_timeout_seconds,
focus_timeout_seconds=payload.focus_timeout_seconds,
+37 -52
View File
@@ -3,6 +3,7 @@ from __future__ import annotations
import json
import sqlite3
import threading
from datetime import datetime, timedelta, timezone
from pathlib import Path
from typing import Any
@@ -217,6 +218,29 @@ class HistoryDB:
).fetchone()
return dict(totals) if totals else {}
def prune_older_than(self, retention_days: int) -> int:
if retention_days < 1:
return 0
cutoff = (datetime.now(timezone.utc) - timedelta(days=retention_days)).isoformat()
with self._lock, self._connect() as conn:
job_ids = [
row[0]
for row in conn.execute(
"""
SELECT job_id
FROM jobs
WHERE created_at < ? AND status IN ('completed', 'failed', 'cancelled')
""",
(cutoff,),
).fetchall()
]
if not job_ids:
return 0
conn.executemany("DELETE FROM job_events WHERE job_id = ?", ((job_id,) for job_id in job_ids))
conn.executemany("DELETE FROM jobs WHERE job_id = ?", ((job_id,) for job_id in job_ids))
conn.commit()
return len(job_ids)
def analytics(self) -> dict[str, Any]:
with self._connect() as conn:
rows = conn.execute(
@@ -227,17 +251,7 @@ class HistoryDB:
"""
).fetchall()
total_jobs = 0
finished_jobs = 0
completed_jobs = 0
failed_jobs = 0
cancelled_jobs = 0
steps_sum = 0
steps_count = 0
cost_sum = 0.0
cost_count = 0
by_category: dict[str, dict[str, Any]] = {}
by_day: dict[str, dict[str, Any]] = {}
def _bucket(target: dict[str, dict[str, Any]], key: str) -> dict[str, Any]:
bucket = target.setdefault(
@@ -258,57 +272,38 @@ class HistoryDB:
return bucket
for row in rows:
total_jobs += 1
status = str(row["status"] or "")
finished = status in _TERMINAL_STATUSES
completed = status == "completed"
objective = str(row["objective"] or "")
category = _objective_category(objective)
created_at = str(row["created_at"] or "")
day = created_at[:10] if len(created_at) >= 10 else created_at or "unknown"
category_bucket = _bucket(by_category, category)
day_bucket = _bucket(by_day, day)
for bucket in (category_bucket, day_bucket):
bucket["total_jobs"] += 1
category_bucket["total_jobs"] += 1
if not finished:
continue
finished_jobs += 1
if completed:
completed_jobs += 1
category_bucket["completed_jobs"] += 1
elif status == "failed":
failed_jobs += 1
category_bucket["failed_jobs"] += 1
elif status == "cancelled":
cancelled_jobs += 1
category_bucket["cancelled_jobs"] += 1
steps = row["steps"]
if steps is not None:
step_value = int(steps)
steps_sum += step_value
steps_count += 1
for bucket in (category_bucket, day_bucket):
bucket["steps_sum"] += step_value
bucket["steps_count"] += 1
category_bucket["steps_sum"] += step_value
category_bucket["steps_count"] += 1
estimated_cost = row["estimated_cost_usd"]
if estimated_cost is not None:
cost_value = float(estimated_cost)
cost_sum += cost_value
cost_count += 1
for bucket in (category_bucket, day_bucket):
bucket["cost_sum"] += cost_value
bucket["cost_count"] += 1
category_bucket["cost_sum"] += cost_value
category_bucket["cost_count"] += 1
for bucket in (category_bucket, day_bucket):
bucket["finished_jobs"] += 1
if completed:
bucket["completed_jobs"] += 1
elif status == "failed":
bucket["failed_jobs"] += 1
elif status == "cancelled":
bucket["cancelled_jobs"] += 1
category_bucket["finished_jobs"] += 1
def _finalize(bucket: dict[str, Any]) -> dict[str, Any]:
finished = bucket["finished_jobs"]
@@ -325,21 +320,10 @@ class HistoryDB:
}
category_rows = [_finalize(bucket) for bucket in by_category.values()]
category_rows.sort(key=lambda item: (-item["success_rate"], item["label"]))
day_rows = [_finalize(bucket) for bucket in by_day.values()]
day_rows.sort(key=lambda item: item["label"])
category_rows.sort(key=lambda item: item["label"])
return {
"total_jobs": total_jobs,
"finished_jobs": finished_jobs,
"completed_jobs": completed_jobs,
"failed_jobs": failed_jobs,
"cancelled_jobs": cancelled_jobs,
"success_rate": round((completed_jobs / finished_jobs) * 100, 2) if finished_jobs else 0.0,
"avg_steps": round(steps_sum / steps_count, 2) if steps_count else None,
"avg_cost_usd": round(cost_sum / cost_count, 6) if cost_count else None,
"by_category": category_rows,
"timeline": day_rows,
}
def _row_to_job(self, row: sqlite3.Row) -> dict[str, Any]:
@@ -348,6 +332,7 @@ class HistoryDB:
disabled_tools = json.loads(row["disabled_tools_json"]) if row["disabled_tools_json"] else []
except Exception:
disabled_tools = []
response = self._parse_response_payload(row["response_json"], row["result"])
return {
"job_id": row["job_id"],
"objective": row["objective"],
@@ -356,8 +341,8 @@ class HistoryDB:
"created_at": row["created_at"],
"started_at": row["started_at"],
"ended_at": row["ended_at"],
"result": row["result"],
"response": self._parse_response_payload(row["response_json"], row["result"]),
"return": response["return"],
"data": response["data"],
"error": row["error"],
"steps": row["steps"],
"cancelled": bool(row["cancelled"]),
+28 -23
View File
@@ -10,7 +10,6 @@ from typing import Any, Callable
from .agent import normalize_disabled_tools
from .config import AppConfig
from .desktop_overlay import DesktopOverlayManager, get_desktop_overlay_manager
from .models import RuntimeOptions
from .runtime import create_openai_client, run_job
from .safety import assess_task_safety
@@ -34,12 +33,10 @@ class JobManager:
config: AppConfig,
db: HistoryDB,
broadcast: Callable[[dict[str, Any]], None] | None = None,
overlay_manager: DesktopOverlayManager | None = None,
) -> None:
self.config = config
self.db = db
self.broadcast = broadcast
self.overlay_manager = overlay_manager or get_desktop_overlay_manager()
self._running: dict[str, _RunningJob] = {}
self._lock = threading.Lock()
@@ -54,7 +51,7 @@ class JobManager:
click_pause: float = 0.10,
reasoning_effort: str = "medium",
screen_context_decay_steps: int = 4,
max_visual_context_images: int = 3,
max_visual_context_images: int | None = None,
native_automation_mode: str = "prefer",
dialog_timeout_seconds: float = 12.0,
focus_timeout_seconds: float = 8.0,
@@ -69,6 +66,12 @@ class JobManager:
created_at = utc_now_iso()
selected_model = (model or self.config.default_model).strip() or self.config.default_model
disabled = normalize_disabled_tools(disabled_tools)
with self._lock:
stale = [key for key, job in self._running.items() if not job.thread.is_alive()]
for key in stale:
self._running.pop(key, None)
if self._running:
raise ValueError("Another ScreenJob run is already active on this desktop.")
self.db.create_job(
job_id=job_id,
objective=objective,
@@ -108,7 +111,11 @@ class JobManager:
"click_pause": click_pause,
"reasoning_effort": reasoning_effort,
"screen_context_decay_steps": screen_context_decay_steps,
"max_visual_context_images": max_visual_context_images,
"max_visual_context_images": (
self.config.max_visual_context_images_default
if max_visual_context_images is None
else max_visual_context_images
),
"native_automation_mode": native_automation_mode,
"dialog_timeout_seconds": dialog_timeout_seconds,
"focus_timeout_seconds": focus_timeout_seconds,
@@ -145,7 +152,7 @@ class JobManager:
click_pause: float,
reasoning_effort: str,
screen_context_decay_steps: int,
max_visual_context_images: int,
max_visual_context_images: int | None,
native_automation_mode: str,
dialog_timeout_seconds: float,
focus_timeout_seconds: float,
@@ -251,7 +258,14 @@ class JobManager:
click_pause=click_pause,
reasoning_effort=reasoning_effort,
screen_context_decay_steps=max(0, int(screen_context_decay_steps)),
max_visual_context_images=max(0, int(max_visual_context_images)),
max_visual_context_images=max(
0,
int(
self.config.max_visual_context_images_default
if max_visual_context_images is None
else max_visual_context_images
),
),
native_automation_mode=str(native_automation_mode or "prefer").strip().lower() or "prefer",
dialog_timeout_seconds=max(0.5, float(dialog_timeout_seconds)),
focus_timeout_seconds=max(0.5, float(focus_timeout_seconds)),
@@ -322,22 +336,14 @@ class JobManager:
"event_type": "job_finished",
"payload": {
"status": status,
"result": result.return_message,
"response": {"return": result.return_message, "data": result.data},
"return": result.return_message,
"data": result.data,
"error": result.error,
"cancelled": result.cancelled,
"usage": result.usage.to_dict(),
},
},
)
if status == "completed":
self.overlay_manager.show_completion(
job_id=job_id,
objective=objective,
return_message=result.return_message,
steps=result.steps,
elapsed_seconds=max(0.0, float(result.ended_at - result.started_at)),
)
with self._lock:
self._running.pop(job_id, None)
@@ -396,10 +402,9 @@ class JobManager:
return self.db.analytics()
def _normalize_job_payload(self, job: dict[str, Any]) -> dict[str, Any]:
response = job.get("response")
if not isinstance(response, dict):
response = {"return": str(job.get("result") or ""), "data": None}
job["response"] = response
job["return"] = str(response.get("return") or "")
job["data"] = response.get("data")
if "return" not in job:
job["return"] = str(job.get("result") or "")
job.setdefault("data", None)
job.pop("response", None)
job.pop("result", None)
return job
+5 -15
View File
@@ -26,22 +26,12 @@
<h2 class="font-semibold">Analytics</h2>
<div id="analyticsMeta" class="text-[11px] text-slate-400"></div>
</div>
<div id="analyticsSummary" class="grid grid-cols-2 md:grid-cols-4 gap-3"></div>
<div class="grid grid-cols-1 xl:grid-cols-2 gap-4">
<div class="bg-slate-900/70 border border-slate-800 rounded-xl p-4 space-y-3">
<div class="flex items-center justify-between gap-3">
<h3 class="font-semibold text-sm">Success by Objective Category</h3>
<div id="analyticsCategorySummary" class="text-[11px] text-slate-400"></div>
</div>
<div id="analyticsCategories" class="space-y-3"></div>
</div>
<div class="bg-slate-900/70 border border-slate-800 rounded-xl p-4 space-y-3">
<div class="flex items-center justify-between gap-3">
<h3 class="font-semibold text-sm">Avg Steps / Cost Over Time</h3>
<div id="analyticsTrendSummary" class="text-[11px] text-slate-400"></div>
</div>
<div id="analyticsTrends" class="space-y-4"></div>
<div class="bg-slate-900/70 border border-slate-800 rounded-xl p-4 space-y-3">
<div class="flex items-center justify-between gap-3">
<h3 class="font-semibold text-sm">Success by Objective Category</h3>
<div id="analyticsCategorySummary" class="text-[11px] text-slate-400"></div>
</div>
<div id="analyticsCategories" class="space-y-3"></div>
</div>
</section>
+1 -116
View File
@@ -18,11 +18,8 @@ const replayNextBtn = document.getElementById("replayNextBtn");
const replaySpeedEl = document.getElementById("replaySpeed");
const replaySeekEl = document.getElementById("replaySeek");
const analyticsMetaEl = document.getElementById("analyticsMeta");
const analyticsSummaryEl = document.getElementById("analyticsSummary");
const analyticsCategorySummaryEl = document.getElementById("analyticsCategorySummary");
const analyticsCategoriesEl = document.getElementById("analyticsCategories");
const analyticsTrendSummaryEl = document.getElementById("analyticsTrendSummary");
const analyticsTrendsEl = document.getElementById("analyticsTrends");
const state = {
token: localStorage.getItem("screenjob_token") || "",
@@ -98,104 +95,10 @@ function formatPercent(value) {
return Number.isFinite(num) ? `${num.toFixed(1)}%` : "—";
}
function formatDateLabel(value) {
const dt = new Date(value);
if (Number.isNaN(dt.getTime())) return String(value || "—");
return dt.toLocaleDateString(undefined, { month: "short", day: "numeric" });
}
function renderMetricCard(label, value) {
return `
<div class="bg-slate-950 border border-slate-800 rounded-xl p-3">
<div class="text-[11px] uppercase tracking-wide text-slate-400">${escapeHtml(label)}</div>
<div class="text-xl font-semibold mt-1">${escapeHtml(value)}</div>
</div>
`;
}
function renderLineChart(title, points, options = {}) {
const color = options.color || "#22d3ee";
const valueLabel = options.valueLabel || "";
const sourcePoints = Array.isArray(points)
? points.filter((point) => Number.isFinite(Number(point.value)))
: [];
if (!sourcePoints.length) {
return `
<div class="rounded-lg border border-slate-800 bg-slate-950/70 p-3">
<div class="flex items-center justify-between gap-3">
<div>
<div class="text-xs text-slate-400">${escapeHtml(title)}</div>
<div class="text-sm text-slate-200 font-semibold">No data yet</div>
</div>
</div>
</div>
`;
}
const width = 640;
const height = 220;
const margin = { top: 20, right: 18, bottom: 34, left: 44 };
const values = sourcePoints.map((point) => Number(point.value));
const minValue = Math.min(...values);
const maxValue = Math.max(...values);
const span = maxValue - minValue || 1;
const chartWidth = width - margin.left - margin.right;
const chartHeight = height - margin.top - margin.bottom;
const xStep = sourcePoints.length > 1 ? chartWidth / (sourcePoints.length - 1) : 0;
const coords = sourcePoints.map((point, index) => ({
x: margin.left + (index * xStep),
y: margin.top + ((maxValue - Number(point.value)) / span) * chartHeight,
}));
const linePath = coords.map((point, index) => `${index === 0 ? "M" : "L"} ${point.x} ${point.y}`).join(" ");
const baseline = height - margin.bottom;
const midIndex = Math.floor(sourcePoints.length / 2);
const xLabels = [
{ index: 0, label: sourcePoints[0].label },
{ index: midIndex, label: sourcePoints[midIndex].label },
{ index: sourcePoints.length - 1, label: sourcePoints[sourcePoints.length - 1].label },
].filter((item, index, array) => item.label && array.findIndex((candidate) => candidate.index === item.index) === index);
const minLabel = options.formatValue ? options.formatValue(minValue) : formatNumber(minValue, 2);
const maxLabel = options.formatValue ? options.formatValue(maxValue) : formatNumber(maxValue, 2);
const latest = sourcePoints[sourcePoints.length - 1];
const latestValue = options.formatValue ? options.formatValue(latest.value) : formatNumber(latest.value, 2);
return `
<div class="rounded-lg border border-slate-800 bg-slate-950/70 p-3 space-y-2">
<div class="flex items-center justify-between gap-3">
<div>
<div class="text-xs text-slate-400">${escapeHtml(title)}</div>
<div class="text-sm text-slate-200 font-semibold">${escapeHtml(latestValue)}${valueLabel ? ` <span class="text-slate-500 font-normal">${escapeHtml(valueLabel)}</span>` : ""}</div>
</div>
<div class="text-[11px] text-slate-400 text-right">
<div>${escapeHtml(sourcePoints.length)} points</div>
<div>${escapeHtml(minLabel)} - ${escapeHtml(maxLabel)}</div>
</div>
</div>
<svg viewBox="0 0 ${width} ${height}" class="w-full h-56">
${Array.from({ length: 4 }, (_, idx) => {
const y = margin.top + (chartHeight / 3) * idx;
return `<line x1="${margin.left}" y1="${y}" x2="${width - margin.right}" y2="${y}" stroke="rgba(51, 65, 85, 0.7)" stroke-width="1" />`;
}).join("")}
<line x1="${margin.left}" y1="${baseline}" x2="${width - margin.right}" y2="${baseline}" stroke="rgba(71, 85, 105, 0.8)" stroke-width="1.5" />
<path d="${linePath}" fill="none" stroke="${color}" stroke-width="3" stroke-linecap="round" stroke-linejoin="round" />
${coords.map((point) => `
<circle cx="${point.x}" cy="${point.y}" r="4.5" fill="${color}" />
`).join("")}
<text x="${margin.left - 8}" y="${margin.top + 4}" text-anchor="end" class="fill-slate-400 text-[10px]">${escapeHtml(maxLabel)}</text>
<text x="${margin.left - 8}" y="${baseline}" text-anchor="end" class="fill-slate-400 text-[10px]">${escapeHtml(minLabel)}</text>
${xLabels.map((item) => `
<text x="${coords[item.index].x}" y="${height - 10}" text-anchor="middle" class="fill-slate-500 text-[10px]">${escapeHtml(formatDateLabel(item.label))}</text>
`).join("")}
</svg>
</div>
`;
}
function renderAnalytics(payload) {
const analytics = payload || {};
const categories = Array.isArray(analytics.by_category) ? analytics.by_category : [];
const timeline = Array.isArray(analytics.timeline) ? analytics.timeline : [];
const finishedCategories = categories.filter((row) => Number(row.finished_jobs || 0) > 0);
if (analyticsMetaEl) {
@@ -204,13 +107,6 @@ function renderAnalytics(payload) {
: "Historical snapshot";
}
analyticsSummaryEl.innerHTML = [
renderMetricCard("Finished Jobs", analytics.finished_jobs || 0),
renderMetricCard("Success Rate", formatPercent(analytics.success_rate)),
renderMetricCard("Avg Steps", formatNumber(analytics.avg_steps, 1)),
renderMetricCard("Avg Cost", formatCurrency(analytics.avg_cost_usd)),
].join("");
analyticsCategorySummaryEl.textContent = finishedCategories.length
? `${finishedCategories.length} categories`
: "No finished jobs yet";
@@ -220,7 +116,6 @@ function renderAnalytics(payload) {
const successRate = Number(row.success_rate || 0);
const completed = Number(row.completed_jobs || 0);
const finished = Number(row.finished_jobs || 0);
const total = Number(row.total_jobs || 0);
const avgSteps = row.avg_steps == null ? "—" : formatNumber(row.avg_steps, 1);
const avgCost = row.avg_cost_usd == null ? "—" : formatCurrency(row.avg_cost_usd);
return `
@@ -228,7 +123,7 @@ function renderAnalytics(payload) {
<div class="flex items-start justify-between gap-3">
<div>
<div class="font-medium">${escapeHtml(row.label || "Other")}</div>
<div class="text-[11px] text-slate-400">${finished} finished · ${completed} completed · ${total} total</div>
<div class="text-[11px] text-slate-400">${finished} finished · ${completed} completed</div>
</div>
<div class="text-right">
<div class="text-base font-semibold">${formatPercent(successRate)}</div>
@@ -252,16 +147,6 @@ function renderAnalytics(payload) {
</div>
`;
}
analyticsTrendSummaryEl.textContent = timeline.length ? `${timeline.length} days` : "No daily data yet";
analyticsTrendsEl.innerHTML = [
renderLineChart("Average steps per day", timeline.map((row) => ({ label: row.label, value: row.avg_steps })), { color: "#38bdf8" }),
renderLineChart("Average cost per day", timeline.map((row) => ({ label: row.label, value: row.avg_cost_usd })), {
color: "#34d399",
valueLabel: "USD",
formatValue: (value) => formatCurrency(value),
}),
].join("");
}
function renderJobs() {
+38 -5
View File
@@ -3,8 +3,9 @@ from __future__ import annotations
import base64
import io
import logging
import shutil
import sys
from datetime import datetime, timezone
from datetime import datetime, timedelta, timezone
from pathlib import Path
from PIL import Image, ImageDraw
@@ -85,23 +86,55 @@ def setup_artifacts(base_dir: Path) -> RunArtifacts:
)
def cleanup_old_run_artifacts(base_dir: Path, retention_days: int) -> int:
if retention_days < 1 or not base_dir.exists() or not base_dir.is_dir():
return 0
cutoff = datetime.now(timezone.utc) - timedelta(days=retention_days)
removed = 0
for child in base_dir.iterdir():
try:
modified_at = datetime.fromtimestamp(child.stat().st_mtime, tz=timezone.utc)
except FileNotFoundError:
continue
if modified_at >= cutoff:
continue
if child.is_dir():
shutil.rmtree(child, ignore_errors=True)
removed += 1
elif child.is_file():
try:
child.unlink()
removed += 1
except FileNotFoundError:
pass
return removed
def setup_logger(log_file: Path, verbose: bool = True) -> logging.Logger:
logger = logging.getLogger("screenjob")
logger_name = f"screenjob.{log_file.parent.parent.name}"
logger = logging.getLogger(logger_name)
logger.setLevel(logging.DEBUG)
logger.handlers.clear()
logger.propagate = False
if logger.handlers:
return logger
stream_level = logging.INFO if verbose else logging.WARNING
stream_handler = logging.StreamHandler(sys.stdout)
stream_handler.setLevel(stream_level)
stream_handler.setFormatter(
logging.Formatter("%(asctime)s | %(levelname)-8s | %(message)s")
logging.Formatter(
"%(asctime)s.%(msecs)03d | %(message)s",
datefmt="%H:%M:%S",
)
)
file_handler = logging.FileHandler(log_file, encoding="utf-8")
file_handler.setLevel(logging.DEBUG)
file_handler.setFormatter(
logging.Formatter(
"%(asctime)s | %(levelname)-8s | %(name)s | %(filename)s:%(lineno)d | %(message)s"
"%(asctime)s.%(msecs)03d | %(levelname)-8s | %(filename)s:%(lineno)d | %(message)s",
datefmt="%Y-%m-%d %H:%M:%S",
)
)
-11
View File
@@ -1,11 +0,0 @@
Option Explicit
Dim shell, fso, scriptDir, psScript, command
Set shell = CreateObject("WScript.Shell")
Set fso = CreateObject("Scripting.FileSystemObject")
scriptDir = fso.GetParentFolderName(WScript.ScriptFullName)
psScript = """" & fso.BuildPath(scriptDir, "screenjob_tray.ps1") & """"
command = "powershell.exe -NoProfile -ExecutionPolicy Bypass -WindowStyle Hidden -STA -File " & psScript
shell.Run command, 0, False
+93 -85
View File
@@ -343,20 +343,20 @@ def test_context_compaction_trigger_and_payload(tmp_path: Path, monkeypatch) ->
agent.step = 4
agent.last_context_compact_step = 0
agent.options.screen_context_decay_steps = 4
agent.recent_tool_summaries = ["step=1 tool=see_screen status=ok"]
agent.recent_tool_summaries = ["step=1 tool=enhance status=ok"]
agent.last_screen_data_url = "data:image/png;base64,abc"
agent.last_screen_meta = {"width": 1280, "height": 720, "path": "C:/tmp/frame.png"}
assert agent._should_compact_context() is True
visual_message = agent._build_visual_message("Current screen", "data:image/png;base64,abc", agent.last_screen_meta)
agent._register_visual_context_message(visual_message, agent.last_screen_meta, tool_name="see_screen")
agent._register_visual_context_message(visual_message, agent.last_screen_meta, tool_name="enhance")
compacted = agent._build_compacted_pending_input("decay")
assert len(compacted) == 2
assert "Context compaction activated due to stale context decay." in compacted[0]["content"][0]["text"]
assert "Open settings app" in compacted[0]["content"][0]["text"]
assert "Treat prior reasoning as stale" in compacted[0]["content"][0]["text"]
assert "Retained visual observations:" in compacted[0]["content"][0]["text"]
assert "do not call see_screen again only because compaction happened" in compacted[0]["content"][0]["text"]
assert "do not ask for another visual just because compaction happened" in compacted[0]["content"][0]["text"]
assert "observe -> decide -> act -> verify" in compacted[0]["content"][0]["text"]
@@ -365,7 +365,7 @@ def test_context_compaction_drops_function_call_outputs_from_rebased_input(tmp_p
agent.objective = "Open settings app"
visual_meta = {"path": "C:/tmp/frame.png"}
visual_message = agent._build_visual_message("Current screen", "data:image/png;base64,abc", visual_meta)
agent._register_visual_context_message(visual_message, visual_meta, tool_name="see_screen")
agent._register_visual_context_message(visual_message, visual_meta, tool_name="enhance")
compacted = agent._build_compacted_pending_input(
"decay",
@@ -390,17 +390,36 @@ def test_visual_context_budget_keeps_only_latest_three_images(tmp_path: Path, mo
"2026-05-30T10:00:01+00:00",
"2026-05-30T10:00:04+00:00",
"2026-05-30T10:00:02+00:00",
"2026-05-30T10:00:05+00:00",
]
for idx, captured_at in enumerate(captured_times):
meta = {"path": f"C:/tmp/frame_{idx}.png", "captured_at": captured_at}
message = agent._build_visual_message(f"frame {idx}", f"data:image/png;base64,{idx}", meta)
agent._register_visual_context_message(message, meta, tool_name="see_screen")
agent._register_visual_context_message(message, meta, tool_name="enhance")
assert agent.visual_context_overflow_pending is True
assert [entry["meta"]["path"] for entry in agent.visual_context_messages] == [
"C:/tmp/frame_3.png",
"C:/tmp/frame_0.png",
"C:/tmp/frame_2.png",
"C:/tmp/frame_4.png",
]
def test_visual_context_budget_does_not_overflow_on_small_headroom(tmp_path: Path, monkeypatch) -> None:
agent = _build_agent(tmp_path, monkeypatch)
agent.options.max_visual_context_images = 3
for idx in range(4):
meta = {"path": f"C:/tmp/frame_{idx}.png"}
message = agent._build_visual_message(f"frame {idx}", f"data:image/png;base64,{idx}", meta)
agent._register_visual_context_message(message, meta, tool_name="enhance")
assert agent.visual_context_overflow_pending is False
assert [entry["meta"]["path"] for entry in agent.visual_context_messages] == [
"C:/tmp/frame_0.png",
"C:/tmp/frame_1.png",
"C:/tmp/frame_2.png",
"C:/tmp/frame_3.png",
]
@@ -419,7 +438,7 @@ def test_compacted_input_uses_latest_visuals_by_capture_time(tmp_path: Path, mon
):
meta = {"path": f"C:/tmp/frame_{idx}.png", "captured_at": captured_at}
message = agent._build_visual_message(f"frame {idx}", f"data:image/png;base64,{idx}", meta)
agent._register_visual_context_message(message, meta, tool_name="see_screen")
agent._register_visual_context_message(message, meta, tool_name="enhance")
compacted = agent._build_compacted_pending_input("visual_budget")
visual_messages = [
@@ -460,33 +479,47 @@ def test_context_compaction_event_includes_visual_budget_reason_and_paths(tmp_pa
assert payload["visual_context_paths"] == ["C:/tmp/1.png", "C:/tmp/2.png", "C:/tmp/3.png"]
def test_context_compaction_log_uses_readable_reason(tmp_path: Path, monkeypatch, caplog) -> None:
agent = _build_agent(tmp_path, monkeypatch)
agent.step = 7
agent.visual_context_messages = [
{"message": {"role": "user", "content": []}, "meta": {"path": "C:/tmp/1.png"}},
]
with caplog.at_level(logging.INFO, logger=agent.logger.name):
agent._emit_context_compacted("visual_budget")
assert "Context compacted at step 7 (reason=visual budget overflow, retained_visuals=1)" in caplog.text
assert "visual_budget" not in caplog.text
def test_observation_loop_blocks_repeated_broad_reobservation(tmp_path: Path, monkeypatch) -> None:
agent = _build_agent(tmp_path, monkeypatch)
agent.step_history = [
{
"step": 21,
"tool_names": ["get_active_window", "see_screen"],
"tool_names": ["get_active_window", "enhance"],
"window_signature": "123|#32770|Save as",
"window_summary": "Save as [#32770]",
"had_visual": True,
},
{
"step": 22,
"tool_names": ["get_active_window", "see_screen"],
"tool_names": ["get_active_window", "enhance"],
"window_signature": "123|#32770|Save as",
"window_summary": "Save as [#32770]",
"had_visual": True,
},
{
"step": 23,
"tool_names": ["get_active_window", "see_screen"],
"tool_names": ["get_active_window", "enhance"],
"window_signature": "123|#32770|Save as",
"window_summary": "Save as [#32770]",
"had_visual": True,
},
]
blocked = agent._dispatch_tool("see_screen", {})
blocked = agent._dispatch_tool("enhance", {})
assert blocked["ok"] is False
assert blocked["blocked"] is True
@@ -508,7 +541,7 @@ def test_observation_loop_counts_sleep_as_non_progress(tmp_path: Path, monkeypat
},
{
"step": 41,
"tool_names": ["see_screen"],
"tool_names": ["enhance"],
"window_signature": "123|#32770|Save as",
"window_summary": "Save as [#32770]",
"had_visual": True,
@@ -540,7 +573,7 @@ def test_observation_loop_counts_sleep_as_non_progress(tmp_path: Path, monkeypat
},
{
"step": 45,
"tool_names": ["see_screen"],
"tool_names": ["enhance"],
"window_signature": "123|#32770|Save as",
"window_summary": "Save as [#32770]",
"had_visual": True,
@@ -548,12 +581,9 @@ def test_observation_loop_counts_sleep_as_non_progress(tmp_path: Path, monkeypat
},
]
blocked = agent._dispatch_tool("see_screen", {})
stable = agent._stable_observation_loop()
assert blocked["ok"] is False
assert blocked["blocked"] is True
assert blocked["blocked_reason"] == "observation_loop"
assert blocked["repeated_steps"] == 3
assert stable is None or stable["window_summary"] == "Save as [#32770]"
def test_observation_loop_treats_focus_window_as_action_progress(tmp_path: Path, monkeypatch) -> None:
@@ -570,7 +600,7 @@ def test_observation_loop_requires_non_empty_window_signature(tmp_path: Path, mo
agent.step_history = [
{
"step": 50,
"tool_names": ["see_screen"],
"tool_names": ["enhance"],
"window_signature": "",
"window_summary": "",
"had_visual": True,
@@ -586,7 +616,7 @@ def test_observation_loop_requires_non_empty_window_signature(tmp_path: Path, mo
},
{
"step": 52,
"tool_names": ["see_screen"],
"tool_names": ["enhance"],
"window_signature": "",
"window_summary": "",
"had_visual": True,
@@ -606,15 +636,11 @@ def test_record_step_history_reuses_last_observed_window_for_visual_only_steps(t
"title": "Settings",
}
agent._record_step_history(["see_screen"], None, "sig-a")
agent._record_step_history(["get_active_window"], None)
agent._record_step_history(["detect_dialog"], None)
agent._record_step_history(["enhance"], None, "sig-a")
stable = agent._stable_observation_loop()
assert stable is not None
assert stable["window_summary"] == "Settings [ApplicationFrameWindow]"
assert stable["repeated_steps"] == 3
entry = agent.step_history[-1]
assert entry["window_summary"] == "Settings [ApplicationFrameWindow]"
assert entry["window_signature"]
def test_repeated_ambiguous_action_requires_verification_and_then_blocks(tmp_path: Path, monkeypatch) -> None:
@@ -622,27 +648,12 @@ def test_repeated_ambiguous_action_requires_verification_and_then_blocks(tmp_pat
type_args = {"text": "repeat me"}
first = agent._dispatch_tool("type", type_args)
second = agent._dispatch_tool("type", type_args)
third = agent._dispatch_tool("type", type_args)
assert first["ok"] is True
assert first["verification_required"] is True
assert first["verification_channels"] == ["enhance", "get_active_window", "see_screen"]
blocked_without_verification = agent._dispatch_tool("type", type_args)
assert blocked_without_verification["blocked"] is True
assert "see_screen" in blocked_without_verification["error"]
assert agent._dispatch_tool("see_screen", {})["ok"] is True
assert agent._dispatch_tool("type", type_args)["ok"] is True
assert agent._dispatch_tool("see_screen", {})["ok"] is True
assert agent._dispatch_tool("type", type_args)["ok"] is True
assert agent._dispatch_tool("see_screen", {})["ok"] is True
blocked_after_retry_budget = agent._dispatch_tool("type", type_args)
assert blocked_after_retry_budget["blocked"] is True
assert "3 time(s) on the same surface" in blocked_after_retry_budget["error"]
assert agent._dispatch_tool("see_screen", {})["ok"] is True
reset_attempt = agent._dispatch_tool("type", type_args)
assert reset_attempt["ok"] is True
assert second["ok"] is True
assert third["ok"] is True
def test_copy_shortcut_prefers_clipboard_verification(tmp_path: Path, monkeypatch) -> None:
@@ -656,7 +667,7 @@ def test_copy_shortcut_prefers_clipboard_verification(tmp_path: Path, monkeypatc
first = agent._dispatch_tool("press_key", {"key": "ctrl+c"})
assert first["ok"] is True
assert first["verification_channels"] == ["clipboard_get"]
assert "verification_channels" not in first
blocked = agent._dispatch_tool("press_key", {"key": "ctrl+c"})
assert blocked["blocked"] is True
@@ -670,6 +681,22 @@ def test_copy_shortcut_prefers_clipboard_verification(tmp_path: Path, monkeypatc
assert second["ok"] is True
def test_sleep_requires_a_previous_tool_call(tmp_path: Path, monkeypatch) -> None:
agent = _build_agent(tmp_path, monkeypatch)
blocked = agent._dispatch_tool("sleep", {"seconds": 0.1})
assert blocked["ok"] is False
assert blocked["blocked"] is True
assert blocked["blocked_reason"] == "sleep_requires_previous_tool_call"
observed = agent._dispatch_tool("get_cursor_position", {})
assert observed["ok"] is True
allowed = agent._dispatch_tool("sleep", {"seconds": 0.1})
assert allowed["ok"] is True
assert allowed["slept_seconds"] == 0.1
def test_execute_command_blocks_unrequested_recursive_file_search(tmp_path: Path, monkeypatch) -> None:
agent = _build_agent(tmp_path, monkeypatch)
agent.objective = "Save the current note in Notepad"
@@ -732,15 +759,15 @@ def test_execute_command_launch_requires_focus_verification(tmp_path: Path, monk
assert first["ok"] is True
assert first["background_launch_assumed"] is True
assert first["focus_change_assumed"] is False
assert first["verification_required"] is True
assert first["verification_channels"] == ["get_active_window", "see_screen"]
assert "verification_required" not in first
assert "verification_channels" not in first
assert called["command"] == "start notepad"
blocked = agent._dispatch_tool("execute_command", {"command": "start notepad"})
assert blocked["blocked"] is True
assert "get_active_window" in blocked["error"]
assert "enhance" in blocked["error"]
observed = agent._dispatch_tool("get_active_window", {})
observed = agent._dispatch_tool("enhance", {})
assert observed["ok"] is True
second = agent._dispatch_tool("execute_command", {"command": "start notepad"})
@@ -750,21 +777,11 @@ def test_execute_command_launch_requires_focus_verification(tmp_path: Path, monk
def test_system_prompt_emphasizes_situational_awareness() -> None:
prompt = agent_module.SYSTEM_PROMPT
assert "Maintain a live mental model" in prompt
assert "classify -> choose control channel -> execute one meaningful transition -> verify" in prompt
assert "First classify, then act." in prompt
assert "Use see_screen at a balanced cadence" in prompt
assert "get_active_window" in prompt
assert "detect_dialog" in prompt
assert "dialog_set_filename" in prompt
assert "list_ui_elements" in prompt
assert "clipboard_get" in prompt
assert "Do not invent new subgoals" in prompt
assert "verify-and-finish" in prompt
assert "Use tools to act" in prompt
assert "observe -> choose the best tool -> make one meaningful move -> verify" in prompt
assert "Do not assume command-launched apps or URLs became foreground" in prompt
assert "Resolve unexpected modals before resuming the old plan" in prompt
assert "data.observed_result" in prompt
assert "Treat command-launched apps or URLs as background" in prompt
assert "#32770" in prompt
assert "secure desktop" in prompt.lower()
def test_observation_loop_prompt_pushes_action_or_finish() -> None:
@@ -786,7 +803,7 @@ def test_finish_likely_prompt_pushes_verification_then_completion() -> None:
assert "objective is likely already satisfied" in prompt
assert "todo-demo.txt - Notepad" in prompt
assert "call see_screen" in prompt
assert "add enhance only if the proof is small or text-heavy" in prompt
assert "then call task_complete" in prompt
assert "Do not reopen menus" in prompt
assert "Prohibited key combos for this run: ctrl+shift+s." in prompt
@@ -800,12 +817,11 @@ def test_initial_action_prompt_reinforces_observation_and_verification() -> None
assert "Identify what changed since the last action or screen capture." in prompt
assert "classify -> choose control channel -> execute one meaningful transition -> verify" in prompt
assert "Prefer native window/dialog/element tools" in prompt
assert "get_active_window plus detect_dialog" in prompt
assert "click then see_screen" in prompt
assert "Do not invent new subgoals" in prompt
assert "Prefer non-visual verification when available" in prompt
assert "wait_for_focus_change" in prompt
assert "#32770 dialogs" in prompt
assert "verify the expected UI or focus change before repeating the same action or chaining another risky action" in prompt
assert "Prohibited key combos for this run: ctrl+shift+s." in prompt
assert "do not re-capture the screen just to reconfirm an obvious large input area" in prompt
assert 'task_complete(return=..., data={"observed_result": ...})' in prompt
@@ -816,7 +832,6 @@ def test_no_tool_prompt_recovers_by_reobserving() -> None:
assert "Recover by re-observing the current desktop state instead of guessing." in prompt
assert "Start by classifying the surface." in prompt
assert "get_active_window" in prompt
assert "detect_dialog" in prompt
assert "clipboard_get" in prompt
assert "native window/dialog/element tools" in prompt
@@ -833,9 +848,7 @@ def test_blocked_action_prompt_reanchors_on_screen_state() -> None:
assert "classify the current surface" in prompt
assert "detect_dialog" in prompt
assert "dialog_set_filename" in prompt
assert "get_active_window" in prompt
assert "get_cursor_position before move_mouse or drag" in prompt
assert "wait_for_focus_change" in prompt
assert "secure desktop or UAC" in prompt
assert "Switch strategy after the fresh classification" in prompt
assert "Prohibited key combos for this run: ctrl+shift+s." in prompt
@@ -848,16 +861,13 @@ def test_tool_schemas_include_completion_and_desktop_awareness_guidance(tmp_path
schemas = {tool["name"]: tool for tool in agent._tool_schemas()}
assert "data.observed_result" in schemas["task_complete"]["description"]
assert "before task_complete" in schemas["see_screen"]["description"]
assert "text-heavy targets" in schemas["enhance"]["description"]
assert "verify copy or cut results" in schemas["clipboard_get"]["description"]
assert "pointer state matters" in schemas["get_cursor_position"]["description"]
assert "verify focus and active app" in schemas["get_active_window"]["description"]
assert "text-heavy" in schemas["enhance"]["description"]
assert "copy or cut" in schemas["clipboard_get"]["description"]
assert "pointer" in schemas["get_cursor_position"]["description"]
assert "foreground focus" in schemas["execute_command"]["description"]
assert "Prohibited for this run: ctrl+shift+s." in schemas["press_key"]["description"]
assert "dialog classification" in schemas["get_active_window"]["description"]
assert "visible top-level windows" in schemas["list_windows"]["description"]
assert "#32770 or picker surface" in schemas["detect_dialog"]["description"]
assert "#32770" in schemas["detect_dialog"]["description"]
assert "filename or path field" in schemas["dialog_set_filename"]["description"]
assert "native child controls" in schemas["list_ui_elements"]["description"]
@@ -868,7 +878,6 @@ def test_tool_schemas_hide_optional_native_tools_when_mode_off(tmp_path: Path, m
schemas = {tool["name"]: tool for tool in agent._tool_schemas()}
assert "get_active_window" in schemas
assert "list_windows" not in schemas
assert "detect_dialog" not in schemas
assert "list_ui_elements" not in schemas
@@ -880,15 +889,14 @@ def test_tool_schemas_hide_windows_only_tools_on_non_windows_host(tmp_path: Path
schemas = {tool["name"]: tool for tool in agent._tool_schemas()}
assert "get_active_window" not in schemas
assert "list_windows" not in schemas
assert "detect_dialog" not in schemas
assert "list_ui_elements" not in schemas
result = agent._dispatch_tool("get_active_window", {})
result = agent._dispatch_tool("list_windows", {})
assert result["ok"] is False
assert result["error"] == "Tool 'get_active_window' is only available on Windows."
assert result["error"] == "Tool 'list_windows' is only available on Windows."
def test_list_windows_returns_structured_surface_metadata(tmp_path: Path, monkeypatch) -> None:
@@ -1041,7 +1049,7 @@ def test_finish_likely_guard_blocks_reopening_menu_after_fresh_verification(tmp_
)
agent.step = 25
verify_result = agent._dispatch_tool("see_screen", {})
verify_result = agent._dispatch_tool("enhance", {})
assert verify_result["ok"] is True
assert verify_result["finish_likely_verification_done"] is True
assert agent.finish_likely_state["fresh_verification_done"] is True
-22
View File
@@ -9,14 +9,6 @@ from src.config import AppConfig
from src.models import AgentResult, RunArtifacts, UsageSummary
class _OverlayRecorder:
def __init__(self) -> None:
self.calls: list[dict[str, Any]] = []
def show_completion(self, **kwargs: Any) -> None:
self.calls.append(kwargs)
def test_cli_emits_structured_return_and_data(monkeypatch: Any, capsys, tmp_path: Path) -> None:
config = AppConfig(
openai_api_key="test_key",
@@ -64,30 +56,16 @@ def test_cli_emits_structured_return_and_data(monkeypatch: Any, capsys, tmp_path
)
return result, artifacts
overlay = _OverlayRecorder()
monkeypatch.setattr(cli_module, "load_app_config", fake_load_app_config)
monkeypatch.setattr(cli_module, "assess_task_safety", fake_assess_task_safety)
monkeypatch.setattr(cli_module, "run_job", fake_run_job)
monkeypatch.setattr(cli_module, "create_openai_client", lambda *_args, **_kwargs: object())
monkeypatch.setattr(cli_module, "get_desktop_overlay_manager", lambda: overlay)
code = cli_module.main(["Open amazon.de"])
assert code == 0
out = capsys.readouterr().out
payload = json.loads(out)
assert overlay.calls == [
{
"job_id": "20260527_000001",
"objective": "Open amazon.de",
"return_message": "Task completed successfully",
"steps": 3,
"elapsed_seconds": 2.5,
}
]
assert payload["response"]["return"] == "Task completed successfully"
assert payload["response"]["data"] == "file1.txt\nfile2.txt"
assert payload["return"] == "Task completed successfully"
assert payload["data"] == "file1.txt\nfile2.txt"
assert captured_kwargs["options"].reasoning_effort == "medium"
-181
View File
@@ -1,181 +0,0 @@
from __future__ import annotations
import types
from collections import deque
from typing import Any
import src.desktop_overlay as desktop_overlay_module
from src.desktop_overlay import CompletionOverlayPayload, DesktopOverlayManager
class _FakeWidget:
def __init__(self, root: "_FakeTk", *, width: int = 360, height: int = 160) -> None:
self._root = root
self._width = width
self._height = height
self._exists = True
self._after_ids: dict[str, tuple[int, Any]] = {}
def withdraw(self) -> None:
return None
def overrideredirect(self, *_args: Any, **_kwargs: Any) -> None:
return None
def attributes(self, *_args: Any, **_kwargs: Any) -> None:
return None
def configure(self, *_args: Any, **_kwargs: Any) -> None:
return None
def pack(self, *_args: Any, **_kwargs: Any) -> None:
return None
def place(self, *_args: Any, **_kwargs: Any) -> None:
return None
def update_idletasks(self) -> None:
return None
def winfo_width(self) -> int:
return self._width
def winfo_height(self) -> int:
return self._height
def winfo_exists(self) -> bool:
return self._exists
def geometry(self, *_args: Any, **_kwargs: Any) -> None:
return None
def deiconify(self) -> None:
return None
def destroy(self) -> None:
self._exists = False
def after(self, delay_ms: int, callback: Any) -> str:
after_id = self._root._schedule(delay_ms, callback)
self._after_ids[after_id] = (delay_ms, callback)
return after_id
def after_cancel(self, after_id: str) -> None:
self._after_ids.pop(after_id, None)
self._root._cancel(after_id)
class _FakeButton(_FakeWidget):
def __init__(self, root: "_FakeTk", command: Any | None = None, **_kwargs: Any) -> None:
super().__init__(root)
self.command = command
class _FakeTk(_FakeWidget):
def __init__(self) -> None:
super().__init__(self)
self._events: deque[tuple[str, int, Any]] = deque()
self._event_seq = 0
self.scheduled_delays: list[int] = []
self.cards: list[_FakeWidget] = []
def withdraw(self) -> None:
return None
def winfo_screenwidth(self) -> int:
return 1920
def _schedule(self, delay_ms: int, callback: Any) -> str:
after_id = f"after-{self._event_seq}"
self._event_seq += 1
self.scheduled_delays.append(delay_ms)
self._events.append((after_id, delay_ms, callback))
return after_id
def _cancel(self, after_id: str) -> None:
self._events = deque(event for event in self._events if event[0] != after_id)
def mainloop(self) -> None:
iterations = 0
while self._events and iterations < 20:
after_id, _delay_ms, callback = self._events.popleft()
iterations += 1
callback()
if any(not card.winfo_exists() for card in self.cards):
return
class _FakeTkModule(types.SimpleNamespace):
def __init__(self, root: _FakeTk) -> None:
super().__init__()
self._root = root
def Tk(self) -> _FakeTk:
return self._root
def Toplevel(self, _root: _FakeTk) -> _FakeWidget:
card = _FakeWidget(self._root)
self._root.cards.append(card)
return card
def Frame(self, root: _FakeWidget, **_kwargs: Any) -> _FakeWidget:
return _FakeWidget(root._root)
def Label(self, root: _FakeWidget, **_kwargs: Any) -> _FakeWidget:
return _FakeWidget(root._root)
def Button(self, root: _FakeWidget, command: Any | None = None, **_kwargs: Any) -> _FakeButton:
return _FakeButton(root._root, command=command)
def test_show_completion_plays_sound_even_without_overlay_thread(monkeypatch: Any) -> None:
manager = DesktopOverlayManager()
calls: list[str] = []
monkeypatch.setattr(desktop_overlay_module.os, "name", "nt", raising=False)
monkeypatch.setattr(manager, "_play_completion_sound", lambda: calls.append("sound"))
monkeypatch.setattr(manager, "_ensure_thread", lambda: False)
manager.show_completion(
job_id="job-123",
objective="Write a report",
return_message="Finished",
steps=5,
elapsed_seconds=12.4,
)
assert calls == ["sound"]
def test_completion_overlay_auto_dismisses(monkeypatch: Any) -> None:
root = _FakeTk()
fake_tk = _FakeTkModule(root)
monkeypatch.setitem(__import__("sys").modules, "tkinter", fake_tk)
manager = DesktopOverlayManager(auto_dismiss_seconds=0.01)
manager._queue.put(
CompletionOverlayPayload(
job_id="job-123",
objective="Write a report",
return_message="Finished",
steps=5,
elapsed_seconds=12.4,
)
)
manager._ui_main()
assert any(delay == 10 for delay in root.scheduled_delays)
assert root.cards[0]._exists is False
def test_play_completion_sound_uses_winsound_message_beep(monkeypatch: Any) -> None:
calls: list[int] = []
fake_winsound = types.SimpleNamespace(MB_ICONASTERISK=64, MessageBeep=lambda value: calls.append(value))
monkeypatch.setattr(desktop_overlay_module.os, "name", "nt", raising=False)
monkeypatch.setattr(desktop_overlay_module, "winsound", fake_winsound)
DesktopOverlayManager()._play_completion_sound()
assert calls == [64]
+15 -90
View File
@@ -7,26 +7,12 @@ from fastapi.testclient import TestClient
import src.server as server_module
from src.config import AppConfig
from src.storage import _objective_category
_TERMINAL_STATUSES = {"completed", "failed", "cancelled"}
def _objective_category(objective: str) -> str:
text = objective.lower()
if any(keyword in text for keyword in ("browser", "website", "amazon", "google", "login", "shopping", "checkout", "orders")):
return "Browser / web"
if any(keyword in text for keyword in ("file", "folder", "directory", "terminal", "shell", "command", "cli", "script", "git", "repo", "install", "pip", "npm")):
return "Files / terminal"
if any(keyword in text for keyword in ("write", "summary", "document", "docs", "report", "email", "message", "readme", "markdown")):
return "Writing / docs"
if any(keyword in text for keyword in ("data", "analysis", "csv", "spreadsheet", "sheet", "table", "chart", "dashboard", "metric", "sql")):
return "Data / analysis"
if any(keyword in text for keyword in ("code", "bug", "fix", "test", "debug", "api", "backend", "frontend", "database", "deploy", "docker", "service", "build")):
return "Development / ops"
return "Other"
class FakeJobManager:
def __init__(self, *, config: AppConfig, db: Any, broadcast: Any = None) -> None:
self.config = config
@@ -94,8 +80,6 @@ class FakeJobManager:
"started_at": created_at,
"ended_at": None,
"steps": 1,
"result": "Running",
"response": {"return": "Running", "data": None},
"return": "Running",
"data": None,
"usage": {
@@ -188,7 +172,6 @@ class FakeJobManager:
def analytics(self) -> dict[str, Any]:
by_category: dict[str, dict[str, Any]] = {}
by_day: dict[str, dict[str, Any]] = {}
def bucket(target: dict[str, dict[str, Any]], key: str) -> dict[str, Any]:
return target.setdefault(
@@ -207,65 +190,28 @@ class FakeJobManager:
},
)
total_jobs = 0
finished_jobs = 0
completed_jobs = 0
failed_jobs = 0
cancelled_jobs = 0
steps_sum = 0
steps_count = 0
cost_sum = 0.0
cost_count = 0
for job in self._jobs.values():
total_jobs += 1
status = str(job.get("status") or "")
finished = status in _TERMINAL_STATUSES
category = _objective_category(str(job.get("objective") or ""))
day = str(job.get("created_at") or "")[:10] or "unknown"
category_bucket = bucket(by_category, category)
day_bucket = bucket(by_day, day)
for item in (category_bucket, day_bucket):
item["total_jobs"] += 1
category_bucket = bucket(by_category, _objective_category(str(job.get("objective") or "")))
category_bucket["total_jobs"] += 1
if not finished:
continue
finished_jobs += 1
category_bucket["finished_jobs"] += 1
if status == "completed":
completed_jobs += 1
category_bucket["completed_jobs"] += 1
elif status == "failed":
failed_jobs += 1
category_bucket["failed_jobs"] += 1
elif status == "cancelled":
cancelled_jobs += 1
category_bucket["cancelled_jobs"] += 1
steps_raw = job.get("steps")
if steps_raw is not None:
steps = int(steps_raw)
steps_sum += steps
steps_count += 1
for item in (category_bucket, day_bucket):
item["steps_sum"] += steps
item["steps_count"] += 1
category_bucket["steps_sum"] += int(steps_raw)
category_bucket["steps_count"] += 1
estimated_cost_raw = (job.get("usage") or {}).get("estimated_cost_usd")
if estimated_cost_raw is not None:
estimated_cost = float(estimated_cost_raw)
cost_sum += estimated_cost
cost_count += 1
for item in (category_bucket, day_bucket):
item["cost_sum"] += estimated_cost
item["cost_count"] += 1
for item in (category_bucket, day_bucket):
item["finished_jobs"] += 1
if status == "completed":
item["completed_jobs"] += 1
elif status == "failed":
item["failed_jobs"] += 1
elif status == "cancelled":
item["cancelled_jobs"] += 1
category_bucket["cost_sum"] += float(estimated_cost_raw)
category_bucket["cost_count"] += 1
def finalize(item: dict[str, Any]) -> dict[str, Any]:
finished = item["finished_jobs"]
@@ -281,18 +227,7 @@ class FakeJobManager:
"avg_cost_usd": round(item["cost_sum"] / item["cost_count"], 6) if item["cost_count"] else None,
}
return {
"total_jobs": total_jobs,
"finished_jobs": finished_jobs,
"completed_jobs": completed_jobs,
"failed_jobs": failed_jobs,
"cancelled_jobs": cancelled_jobs,
"success_rate": round((completed_jobs / finished_jobs) * 100, 2) if finished_jobs else 0.0,
"avg_steps": round(steps_sum / steps_count, 2) if steps_count else None,
"avg_cost_usd": round(cost_sum / cost_count, 6) if cost_count else None,
"by_category": sorted((finalize(item) for item in by_category.values()), key=lambda item: (-item["success_rate"], item["label"])),
"timeline": sorted((finalize(item) for item in by_day.values()), key=lambda item: item["label"]),
}
return {"by_category": sorted((finalize(item) for item in by_category.values()), key=lambda item: item["label"])}
def _build_app(tmp_path: Path, monkeypatch: Any, disable_ui: bool = False):
@@ -352,8 +287,8 @@ def test_create_job_returns_only_job_id_and_defaults_model(tmp_path: Path, monke
status_res = client.get(f"/api/jobs/{job_id}/status", headers=headers)
assert status_res.status_code == 200
assert status_res.json()["job_id"] == job_id
assert status_res.json()["response"]["return"] == "Running"
assert "data" in status_res.json()["response"]
assert status_res.json()["return"] == "Running"
assert status_res.json()["data"] is None
def test_create_job_rejects_invalid_disabled_tool_names(tmp_path: Path, monkeypatch: Any) -> None:
@@ -459,7 +394,7 @@ def test_replay_endpoint_skips_visual_paths_outside_artifacts(tmp_path: Path, mo
assert payload["total_frames"] == 1
def test_analytics_endpoint_groups_by_category_and_time(tmp_path: Path, monkeypatch: Any) -> None:
def test_analytics_endpoint_groups_by_category(tmp_path: Path, monkeypatch: Any) -> None:
app, _ = _build_app(tmp_path, monkeypatch, disable_ui=False)
manager = app.state.manager
client = TestClient(app)
@@ -495,14 +430,6 @@ def test_analytics_endpoint_groups_by_category_and_time(tmp_path: Path, monkeypa
assert analytics.status_code == 200
payload = analytics.json()
assert payload["total_jobs"] == 3
assert payload["finished_jobs"] == 3
assert payload["completed_jobs"] == 2
assert payload["failed_jobs"] == 1
assert payload["success_rate"] == 66.67
assert payload["avg_steps"] == 6.67
assert payload["avg_cost_usd"] == 0.136667
browser = next(row for row in payload["by_category"] if row["label"] == "Browser / web")
terminal = next(row for row in payload["by_category"] if row["label"] == "Files / terminal")
assert browser["finished_jobs"] == 2
@@ -510,8 +437,6 @@ def test_analytics_endpoint_groups_by_category_and_time(tmp_path: Path, monkeypa
assert browser["avg_steps"] == 5.0
assert terminal["success_rate"] == 100.0
assert [row["label"] for row in payload["timeline"]] == ["2026-05-27", "2026-05-28"]
def test_ui_toggle(tmp_path: Path, monkeypatch: Any) -> None:
app_enabled, _ = _build_app(tmp_path / "enabled", monkeypatch, disable_ui=False)
+32 -14
View File
@@ -37,8 +37,8 @@ def test_history_db_job_and_events_roundtrip(tmp_path: Path) -> None:
assert job["status"] == "completed"
assert job["model"] == "gpt-5.4-mini"
assert job["disabled_tools"] == ["click"]
assert job["response"]["return"] == "Done"
assert job["response"]["data"]["files"] == ["a.txt", "b.txt"]
assert job["return"] == "Done"
assert job["data"]["files"] == ["a.txt", "b.txt"]
assert job["usage"]["estimated_cost_usd"] == 0.1234
events = db.get_job_events(job_id, limit=10)
@@ -70,11 +70,11 @@ def test_storage_response_fallback_uses_result_when_json_missing(tmp_path: Path)
db.update_job(job_id, status="completed", result="Legacy result string")
job = db.get_job(job_id)
assert job is not None
assert job["response"]["return"] == "Legacy result string"
assert job["response"]["data"] is None
assert job["return"] == "Legacy result string"
assert job["data"] is None
def test_history_db_analytics_groups_by_category_and_day(tmp_path: Path) -> None:
def test_history_db_analytics_groups_by_category(tmp_path: Path) -> None:
db = HistoryDB(tmp_path / "screenjob_test_analytics.db")
db.create_job(
@@ -108,14 +108,6 @@ def test_history_db_analytics_groups_by_category_and_day(tmp_path: Path) -> None
db.update_job("job_terminal_ok", status="completed", steps=10, estimated_cost_usd=0.05)
analytics = db.analytics()
assert analytics["total_jobs"] == 3
assert analytics["finished_jobs"] == 3
assert analytics["completed_jobs"] == 2
assert analytics["failed_jobs"] == 1
assert analytics["success_rate"] == 66.67
assert analytics["avg_steps"] == 6.67
assert analytics["avg_cost_usd"] == 0.136667
browser = next(row for row in analytics["by_category"] if row["label"] == "Browser / web")
terminal = next(row for row in analytics["by_category"] if row["label"] == "Files / terminal")
assert browser["finished_jobs"] == 2
@@ -123,4 +115,30 @@ def test_history_db_analytics_groups_by_category_and_day(tmp_path: Path) -> None
assert browser["avg_steps"] == 5.0
assert terminal["success_rate"] == 100.0
assert [row["label"] for row in analytics["timeline"]] == ["2026-05-27", "2026-05-28"]
def test_prune_older_than_removes_old_terminal_jobs(tmp_path: Path) -> None:
db = HistoryDB(tmp_path / "screenjob_prune.db")
db.create_job(
job_id="job_old",
objective="Old",
model="gpt-5.4-mini",
created_at="2000-01-01T00:00:00+00:00",
safety_override=False,
disabled_tools=[],
)
db.update_job("job_old", status="completed")
db.add_event(job_id="job_old", ts="2000-01-01T00:00:01+00:00", step=1, event_type="done", payload={})
db.create_job(
job_id="job_running",
objective="Running",
model="gpt-5.4-mini",
created_at="2000-01-01T00:00:00+00:00",
safety_override=False,
disabled_tools=[],
)
db.update_job("job_running", status="running")
assert db.prune_older_than(7) == 1
assert db.get_job("job_old") is None
assert db.get_job("job_running") is not None
+48 -37
View File
@@ -4,6 +4,8 @@ import threading
from pathlib import Path
from typing import Any
import pytest
import src.task_manager as task_manager_module
from src.config import AppConfig
from src.models import AgentResult, RunArtifacts, UsageSummary
@@ -11,15 +13,7 @@ from src.storage import HistoryDB
from src.task_manager import JobManager
class _OverlayRecorder:
def __init__(self) -> None:
self.calls: list[dict[str, Any]] = []
def show_completion(self, **kwargs: Any) -> None:
self.calls.append(kwargs)
def _build_manager(tmp_path: Path, overlay_manager: _OverlayRecorder) -> tuple[JobManager, HistoryDB, AppConfig]:
def _build_manager(tmp_path: Path) -> tuple[JobManager, HistoryDB, AppConfig]:
config = AppConfig(
openai_api_key="test-key",
screenjob_token="test-token",
@@ -32,7 +26,7 @@ def _build_manager(tmp_path: Path, overlay_manager: _OverlayRecorder) -> tuple[J
db_path=tmp_path / "screenjob.db",
)
db = HistoryDB(config.db_path)
manager = JobManager(config=config, db=db, overlay_manager=overlay_manager)
manager = JobManager(config=config, db=db)
return manager, db, config
@@ -59,10 +53,9 @@ def _create_job(db: HistoryDB, job_id: str, objective: str) -> None:
)
def test_completed_job_triggers_desktop_overlay(tmp_path: Path, monkeypatch) -> None:
overlay = _OverlayRecorder()
manager, db, _config = _build_manager(tmp_path, overlay)
job_id = "job_overlay_complete"
def test_completed_job_updates_status(tmp_path: Path, monkeypatch) -> None:
manager, db, _config = _build_manager(tmp_path)
job_id = "job_complete"
objective = "Save todo-demo.txt in Documents"
_create_job(db, job_id, objective)
@@ -101,23 +94,16 @@ def test_completed_job_triggers_desktop_overlay(tmp_path: Path, monkeypatch) ->
cancel_event=threading.Event(),
)
assert overlay.calls == [
{
"job_id": job_id,
"objective": objective,
"return_message": "Saved todo-demo.txt",
"steps": 11,
"elapsed_seconds": 12.599999999999994,
}
]
assert db.get_job(job_id)["status"] == "completed"
job = db.get_job(job_id)
assert job is not None
assert job["status"] == "completed"
assert job["return"] == "Saved todo-demo.txt"
def test_non_completed_jobs_do_not_trigger_desktop_overlay(tmp_path: Path, monkeypatch) -> None:
overlay = _OverlayRecorder()
manager, db, _config = _build_manager(tmp_path, overlay)
def test_non_completed_jobs_are_recorded(tmp_path: Path, monkeypatch) -> None:
manager, db, _config = _build_manager(tmp_path)
failed_job_id = "job_overlay_failed"
failed_job_id = "job_failed"
_create_job(db, failed_job_id, "Fail intentionally")
failed_result = AgentResult(
completed=False,
@@ -131,7 +117,6 @@ def test_non_completed_jobs_do_not_trigger_desktop_overlay(tmp_path: Path, monke
error="Failure",
)
monkeypatch.setattr(task_manager_module, "run_job", lambda **_kwargs: (failed_result, _artifacts(tmp_path)))
manager._execute_job(
job_id=failed_job_id,
objective="Fail intentionally",
@@ -155,7 +140,7 @@ def test_non_completed_jobs_do_not_trigger_desktop_overlay(tmp_path: Path, monke
cancel_event=threading.Event(),
)
cancelled_job_id = "job_overlay_cancelled"
cancelled_job_id = "job_cancelled"
_create_job(db, cancelled_job_id, "Cancel intentionally")
cancelled_result = AgentResult(
completed=False,
@@ -170,7 +155,6 @@ def test_non_completed_jobs_do_not_trigger_desktop_overlay(tmp_path: Path, monke
cancelled=True,
)
monkeypatch.setattr(task_manager_module, "run_job", lambda **_kwargs: (cancelled_result, _artifacts(tmp_path)))
manager._execute_job(
job_id=cancelled_job_id,
objective="Cancel intentionally",
@@ -194,13 +178,13 @@ def test_non_completed_jobs_do_not_trigger_desktop_overlay(tmp_path: Path, monke
cancel_event=threading.Event(),
)
assert overlay.calls == []
assert db.get_job(failed_job_id)["status"] == "failed"
assert db.get_job(cancelled_job_id)["status"] == "cancelled"
def test_rejected_job_does_not_trigger_desktop_overlay(tmp_path: Path, monkeypatch) -> None:
overlay = _OverlayRecorder()
manager, db, _config = _build_manager(tmp_path, overlay)
job_id = "job_overlay_rejected"
def test_rejected_job_is_recorded(tmp_path: Path, monkeypatch) -> None:
manager, db, _config = _build_manager(tmp_path)
job_id = "job_rejected"
_create_job(db, job_id, "Do something unsafe")
monkeypatch.setattr(task_manager_module, "create_openai_client", lambda *_args, **_kwargs: object())
@@ -233,6 +217,33 @@ def test_rejected_job_does_not_trigger_desktop_overlay(tmp_path: Path, monkeypat
cancel_event=threading.Event(),
)
assert overlay.calls == []
events = db.get_job_events(job_id)
assert events[-1]["event_type"] == "job_rejected"
def test_submit_job_rejects_when_another_run_is_active(tmp_path: Path) -> None:
manager, _db, _config = _build_manager(tmp_path)
ready = threading.Event()
release = threading.Event()
def _hold() -> None:
ready.set()
release.wait()
active_thread = threading.Thread(target=_hold)
active_thread.start()
ready.wait(1)
manager._running["job_active"] = task_manager_module._RunningJob(
thread=active_thread,
cancel_event=threading.Event(),
started_at="2026-05-30T12:00:00+00:00",
objective="Active",
model="gpt-5.4-mini",
)
try:
with pytest.raises(ValueError, match="already active"):
manager.submit_job(objective="Second run")
finally:
release.set()
active_thread.join(timeout=1)
-53
View File
@@ -1,53 +0,0 @@
[CmdletBinding()]
param(
[ValidateSet("start", "stop", "restart")]
[string]$Action,
[string]$ServiceName = "ScreenJobBackend"
)
Set-StrictMode -Version Latest
$ErrorActionPreference = "Stop"
function Wait-ForStatus {
param(
[Parameter(Mandatory = $true)]$Service,
[Parameter(Mandatory = $true)][System.ServiceProcess.ServiceControllerStatus]$TargetStatus,
[int]$TimeoutSeconds = 20
)
$deadline = (Get-Date).AddSeconds($TimeoutSeconds)
while ((Get-Date) -lt $deadline) {
$Service.Refresh()
if ($Service.Status -eq $TargetStatus) {
return
}
Start-Sleep -Milliseconds 350
}
throw "Timed out waiting for service '$($Service.ServiceName)' to reach status '$TargetStatus'."
}
$service = Get-Service -Name $ServiceName -ErrorAction Stop
switch ($Action) {
"start" {
if ($service.Status -ne [System.ServiceProcess.ServiceControllerStatus]::Running) {
Start-Service -Name $ServiceName -ErrorAction Stop
Wait-ForStatus -Service $service -TargetStatus ([System.ServiceProcess.ServiceControllerStatus]::Running)
}
}
"stop" {
if ($service.Status -ne [System.ServiceProcess.ServiceControllerStatus]::Stopped) {
Stop-Service -Name $ServiceName -Force -ErrorAction Stop
Wait-ForStatus -Service $service -TargetStatus ([System.ServiceProcess.ServiceControllerStatus]::Stopped)
}
}
"restart" {
if ($service.Status -eq [System.ServiceProcess.ServiceControllerStatus]::Running) {
Restart-Service -Name $ServiceName -Force -ErrorAction Stop
} else {
Start-Service -Name $ServiceName -ErrorAction Stop
}
Wait-ForStatus -Service $service -TargetStatus ([System.ServiceProcess.ServiceControllerStatus]::Running)
}
}