Decoding Pedestrian Crossing Intentions from Egocentric Vision via Vision-Language Models

Abstract

Egocentric vision captures pedestrian visual perception and behavior from a first-person perspective, offering fine-grained insights crucial for pedestrian intention modeling. However, its application in traffic safety remains largely unexplored. We reformulate pedestrian crossing intention prediction as a Visual Question Answering (VQA) task, leveraging the pre-trained knowledge and reasoning abilities of Vision-Language Models (VLMs). We first benchmark two state-of-the-art VLMs in a zero-shot setting, finding that they achieve moderate gains over random guessing in accuracy but exhibit limited higher-level reasoning (e.g., vehicle dynamics). Built upon this observation, we employ parameter-efficient fine-tuning to adapt VLMs to the task. Our results show that the adapted models significantly outperform zero-shot approaches and achieve a 9% relative improvement in accuracy over a specialized transformer-based baseline. Finally, we demonstrate that integrating dynamic eye-gaze signals and personal attributes further boosts predictive accuracy, establishing a new state-of-the-art for egocentric intent decoding.