Events

Anticipating the Unseen and Unheard for Embodied Perception

Computer vision has seen major success in learning to recognize objects from massive “disembodied” Web photo collections labeled by human annotators. Yet cognitive science tells us that perception develops in the context of acting the world—and without intensive supervision. Meanwhile, many realistic vision tasks require not only categorizing a well-composed human-taken photo, but also actively deciding where to look in the first place. In the context of these challenges, we are exploring how perception benefits from anticipating the sights and sounds an agent will experience as a function of its own actions. We introduce methods to learn predictive representations from unlabeled video accompanied by multi-modal sensory data like egomotion and sound. Using these representations, we demonstrate the impact for low-shot visual recognition and visually-guided audio source separation. Moving from passively captured video to agents that control their own first-person cameras, we investigate how agents can learn to move intelligently to acquire visual observations