À propos
November 2026 preprint from Autodesk Research combining computer-use agents and GenUI to build persistent interface patches over existing applications. It recovers controls from rendered pixels, generates new interaction surfaces, and synchronizes them with the underlying app without requiring source code, APIs, or accessibility data.
Résumé
Why Interface Quilt Adds New Signal:
1. Retrofitted interfaces: Demonstrates task-specific controls, marking menus, and other overlays across PowerPoint, Affinity, Fusion, Slack, and Overleaf
2. Build/runtime separation: Agents explore the app and generate patches; a surface, controller, and OS daemon coordinate runtime input and state
3. Reusable logic: Separates patch behavior from application-specific bindings
4. Research boundary: Under-review preprint with demonstrations; build-test iterations can take tens of minutes, hidden state limits synchronization, and some input paths interrupt the user
1. Retrofitted interfaces: Demonstrates task-specific controls, marking menus, and other overlays across PowerPoint, Affinity, Fusion, Slack, and Overleaf
2. Build/runtime separation: Agents explore the app and generate patches; a surface, controller, and OS daemon coordinate runtime input and state
3. Reusable logic: Separates patch behavior from application-specific bindings
4. Research boundary: Under-review preprint with demonstrations; build-test iterations can take tens of minutes, hidden state limits synchronization, and some input paths interrupt the user
Tags
generative-uicomputer-use-agentsinterface-patchesmalleable-uiruntime-ui