claw-ideas/ideas/mobile-integration/screen-context-assistant.md

Screen-Aware Context Assistant

Provide the agent with context from what you're doing on your phone.

Overview

Uses screen recordings to help the agent understand your mobile workflow and provide assistance.

Features

• Workflow Troubleshooting: Record a bug or a confusing UI with screen_record for the agent to analyze.
• Mobile Tutorial Generator: Agent watches your screen_record and generates a step-by-step guide.
• Contextual Q&A: Ask questions about an app or website currently on your screen.
• Data Extraction: Extracts text or data from mobile apps that don't have APIs.