space/clickthrough
Archived
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Improve OCR reliability on dense real-world UIs #18

New Issue
Closed
opened 2026-05-01 17:11:55 +02:00 by luna · 0 comments
luna commented 2026-05-01 17:11:55 +02:00
Collaborator
Copy Link

Problem

During live use, OCR on real app UIs was noisy and inconsistent on dense layouts like Amazon checkout pages and OBS control strips. Full-screen OCR frequently returned garbage, partial labels, or missed the one actionable button entirely.

Observed pain

  • Full-page OCR often missed important buttons like Buy Now
  • Dense sidebars / control strips produced fragmented text
  • The same screen sometimes required repeated manual re-cropping before OCR became useful

Expected improvement

Improve OCR quality and targeting for UI workflows, especially:

  • button-heavy sidebars
  • shopping / checkout pages
  • streaming / recording apps like OBS
  • small control clusters in bottom/right panels

Ideas

  • Better preprocessing for OCR on UI screenshots
  • Optional OCR tuned for UI text rather than document text
  • Region-first OCR helpers that make tight crops easier and more reliable
  • Better handling of small buttons / low-contrast labels
## Problem During live use, OCR on real app UIs was noisy and inconsistent on dense layouts like Amazon checkout pages and OBS control strips. Full-screen OCR frequently returned garbage, partial labels, or missed the one actionable button entirely. ## Observed pain - Full-page OCR often missed important buttons like `Buy Now` - Dense sidebars / control strips produced fragmented text - The same screen sometimes required repeated manual re-cropping before OCR became useful ## Expected improvement Improve OCR quality and targeting for UI workflows, especially: - button-heavy sidebars - shopping / checkout pages - streaming / recording apps like OBS - small control clusters in bottom/right panels ## Ideas - Better preprocessing for OCR on UI screenshots - Optional OCR tuned for UI text rather than document text - Region-first OCR helpers that make tight crops easier and more reliable - Better handling of small buttons / low-contrast labels
luna closed this issue 2026-05-01 17:14:49 +02:00
This repo is archived. You cannot comment on issues.
No Branch/Tag Specified
Branches Tags
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
luna space
No Assignees
1 Participants
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: space/clickthrough#18
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?