Look2React: Making VR NPCs Come Alive with Dynamic Vision-Guided React...

📅 Session: Eye and motion tracking 📖 Title: Look2React: Making VR NPCs Come Alive with Dynamic Vision-Guided Reactions 👨‍🏫 Authors: Ritik Vatsal: University of British Columbia; Xincheng Huang: University of British Columbia; Robert Xiao: University of British Columbia 📜 Abstract: A central promise of VR games is the increased control players have through pose and body language. However, many non-player character (NPC) systems fail to respond convincingly to these poses, user intent, and situational context, limiting immersion. We present Look2React, an interaction system that captures what NPCs see, using a vision-based reasoning model to select pose and text responses, allowing the NPCs to react dynamically. Through our user study, we show that Look2React boosts engagement, encouraging more frequent NPC interactions. Participants reported higher satisfaction, stronger social presence, and NPCs with greater depth and personality --- making them feel more human. https://ieeevr.org/2026/