Skip to main content

ADR-0010: Desktop Screenshot-to-Narrative (Vision)

Status​

Proposed / exploring. Captured during ideation on 2026-07-04. Not yet decided; revisit after the core loop (SPEC-0001..0003) is standing.

Context and Problem Statement​

The embedded-browser idea (ADR-0009) works for web projects, but native desktop apps can't be dropped into a WKWebView. Yet the same need exists: review the running product, not just its source.

The idea: the human takes a screenshot of the running desktop app; a vision model views it, understands what's on screen, and packages a narrative that ties the screenshot to the relevant source files — a reverse of Tour mode (ADR-0008). The human can then mark up the screenshot ("this label is wrong", "this state shouldn't be reachable"), and the narrative gives the dispatched agent the map from pixels to code.

Sketch​

  • The user captures (or drags in) a screenshot of the running app, optionally with a region selection or annotation.
  • A vision-capable model analyzes the screenshot against the repository and emits a narrative: what the screen is, which UI elements are visible, and the candidate source files/symbols responsible for them (with confidence).
  • The narrative renders like an (inverted) Tour (ADR-0008): stops link screenshot regions to file+range. The human marks up regions; markup + narrative compile into a Brief (ADR-0006) whose anchors are (screenshot-region → source) pairs.
  • Optional: pair with the OS screenshot/accessibility APIs to enrich the mapping (element roles, labels) beyond raw pixels.

Why it fits​

  • Good, because it brings running-artifact review to native apps, where the browser approach can't reach.
  • Good, because it inverts machinery we already need: Tour mode goes code→reading order; this goes screen→code.
  • Good, because bring-your-own-model already assumes an OpenAI-compatible endpoint; this just requires a vision-capable one.

Open questions​

  • Screenshot-region → source mapping quality without a DOM; how much to lean on accessibility metadata vs. pure vision.
  • Whether capture is manual (user screenshots) or Stet-driven (drive the app and capture) — the latter overlaps with computer-use tooling.
  • Handling apps that aren't the project under review (scope/safety).
  • Brief anchor representation for screenshot regions.

Consequences (tentative)​

  • Requires a vision model path (endpoint capability detection) alongside the text path.
  • Adds a screenshot canvas mode + region annotation, sharing the annotation layer with ADR-0009.
  • Expands the Narration/Brief schema (ADR-0008, SPEC-0002) with screen-region anchors.

More Information​

Sibling idea for web targets: ADR-0009 (embedded-browser markup). Shared theme: review the running artifact, not only the source.