소개
November 2026 preprint generating both spatial controls and the connector code for physical devices when a Quest 3 wearer requests them. A coding agent tests connections over hand-written MIDI, Bluetooth, and key-press transports, then returns AR widgets, a per-frame hand program, or both.
요약
Generating Controls and Their Connections:
1. Bound outputs: Generated widgets invoke actions declared by a running device connector
2. Local interaction: Hand programs execute every frame without further model calls; runtime checks constrain their outputs
3. Measured setup: Nine of ten requests reached ready in one author-operated session; median setup was 32.5 seconds, excluding speech transcription
4. Research boundary: Readiness establishes transport acknowledgements, not measured physical effects; targets used prior device knowledge and prepared transports, and laptop connectors are not sandboxed
[Author's code, emulator, and recorded demonstrations](https://github.com/Apolotary/controlanything).
1. Bound outputs: Generated widgets invoke actions declared by a running device connector
2. Local interaction: Hand programs execute every frame without further model calls; runtime checks constrain their outputs
3. Measured setup: Nine of ten requests reached ready in one author-operated session; median setup was 32.5 seconds, excluding speech transcription
4. Research boundary: Readiness establishes transport acknowledgements, not measured physical effects; targets used prior device knowledge and prepared transports, and laptop connectors are not sandboxed
[Author's code, emulator, and recorded demonstrations](https://github.com/Apolotary/controlanything).
태그
generative-uimixed-realitydevice-integrationhand-trackingruntime-ui