List рдкрд░ рд╡рд╛рдкрд╕ рдЬрд╛рдПрдБ
ЁЯУДPapers

AesCode: Aesthetic Code Generation with Decoupled Cross-Modal Rewards

Gao X. et al.
2026-10-13

рдкрд░рд┐рдЪрдп

October 2026 paper using generated reference images to guide editable HTML/CSS generation for dashboards, webpages, slides, and posters. AesCode keeps the prompt authoritative for content, represents the design as a graph, and trains with separate programmatic checks and visual rewards.

рд╕рд╛рд░рд╛рдВрд╢

Why AesCode Adds New Signal:

1. Guidance without copying errors: Learns layout and style from generated images while grounding text and data in the prompt
2. Attributable rewards: Routes design-graph properties to deterministic verifiers or graph-specific visual questions, then normalizes reward channels separately
3. Separated evaluation: Reports visual quality and verifiable correctness on 300 infographic prompts with three generations each and no selection
4. Research boundary: Evaluates visual artifacts rather than runtime interaction; complex relationships and SVG icons remain difficult, and reference images can contain hallucinations

[Official project, examples, code, and model links](https://microsoft.github.io/AesCode/).

Tags

ui-generationvisual-guidancedesign-graphsreinforcement-learningcode-verification
рдореВрд▓ рд▓реЗрдЦ рдкрдврд╝реЗрдВ

рдорд┐рд▓рддреЗ-рдЬреБрд▓рддреЗ рд▓реЗрдЦ

ЁЯУДPapers

ControlAnything: Generative UI and On-Demand Device Integration in Mixed Reality

November 2026 preprint generating both spatial controls and the connector code for physical devices when a Quest 3 wearer requests them. A coding agent tests connections over hand-written MIDI, Bluetooth, and key-press transports, then returns AR widgets, a per-frame hand program, or both.

generative-uimixed-realitydevice-integration
рд▓реЗрдЦрдХ Bektur Ryskeldiev ┬╖ 2026-11-05
рдФрд░ рдкрдврд╝реЗрдВ
ЁЯУДPapers

Interface Quilt: Modifying Applications with Computer Use Agents and Generative UI

November 2026 preprint from Autodesk Research combining computer-use agents and GenUI to build persistent interface patches over existing applications. It recovers controls from rendered pixels, generates new interaction surfaces, and synchronizes them with the underlying app without requiring source code, APIs, or accessibility data.

generative-uicomputer-use-agentsinterface-patches
рд▓реЗрдЦрдХ Qiu L. et al. ┬╖ 2026-11-01
рдФрд░ рдкрдврд╝реЗрдВ
ЁЯУДPapers

ABRA: Agent-Bound UI for Human-Agent Communication beyond Chat with Reusable Views

October 2026 EMNLP system demonstration separating reusable graph, table, text, and display views from task-specific interaction meanings. ABRA publishes these bindings as shared canvas state so users and other agents can inspect and invoke task-specific actions.

generative-uishared-canvasesinteraction-semantics
рд▓реЗрдЦрдХ Kim S. G. and Kim H. ┬╖ 2026-10-22
рдФрд░ рдкрдврд╝реЗрдВ

Generative UI рдФрд░ Intelligent UI resources рдХрд╛ curated рд╕рдВрдЧреНрд░рд╣ред

рдмрдирд╛рдпрд╛ рдЧрдпрд╛ тЭдя╕П community рджреНрд╡рд╛рд░рд╛