Sindhu B. Hegde

Hi! I am a second year PhD student in the Visual Geometry Group (VGG) at the University of Oxford, supervised by Prof. Andrew Zisserman. My research is in Computer Vision, particularly in multimodal learning, video understanding and self-supervised learning.

Prior to joining Oxford, I worked as a Lead Data Scientist @ Verisk Analytics. Before that, I pursued Masters’ by Research (MS) at Centre for Visual Information Technology (CVIT), IIIT Hyderabad supervised by Prof. C V Jawahar (IIIT-H) and Prof. Vinay Namboodiri (University of Bath, UK). My Masters’ research focused on exploiting the redundancies in vision and speech modalities for cross-modal generation.

Research interests: Computer Vision, Machine Learning, Deep Learning, Video Understanding, Multi-modal Learning: Vision + Speech/Language

News [Archive]

Sep 2023	Our paper on GestSync: Determining who is speaking without a talking head accepted to BMVC 2023 (ORAL)
Jul 2023	Participated in the International Computer Vision Summer School (ICVSS)) at Sicily, Italy. Had an eincredible experience of learning from some of the most distinguished computer vision experts!
Oct 2022	Joined the Visual Geometry Group (VGG) at the University of Oxford as a PhD student with Prof. Andrew Zisserman
Jul 2022	2 papers accepted to ACM-MM 2022 1] Talking-Face Video Upsampling 2] Lip-to-Speech Synthesis
May 2022	Successfully defended MS thesis Thesis: Exploiting Cross-Modal Redundancy for Audio-Visual Generation

Recent papers [Full list]

BMVC

GestSync: Determining who is speaking without a talking head

Hegde, Sindhu, and Zisserman, Andrew

In British Machine Vision Conference (BMVC) 2023

Abs PDF Code Website

In this paper we introduce a new synchronisation task, Gesture-Sync: determining if a person’s gestures are correlated with their speech or not. In comparison to Lip-Sync, Gesture-Sync is far more challenging as there is a far looser relationship between the voice and body movement than there is between voice and lip motion. We introduce a dual-encoder model for this task, and compare a number of input representations including RGB frames, keypoint images, and keypoint vectors, assessing their performance and advantages. We show that the model can be trained using self-supervised learning alone, and evaluate its performance on the LRS3 dataset. Finally, we demonstrate applications of Gesture-Sync for audio-visual synchronisation, and in determining who is the speaker in a crowd, without seeing their faces.
ACM-MM

Extreme-scale Talking-Face Video Upsampling with Audio-Visual Priors

Hegde, Sindhu, Mukhopadhyay, Rudrabha, Namboodiri, Vinay P, and Jawahar, CV

In Proceedings of the 30th ACM International Conference on Multimedia (MM’22) 2022

Abs PDF Code Website

In this paper, we explore an interesting question of what can be obtained from an 8×8 pixel video sequence. Surprisingly, it turns out to be quite a lot. We show that when we process this 8x8 video with the right set of audio and image priors, we can obtain a full-length, 256x256 video. We achieve this 32x scaling of an extremely low-resolution input using our novel audio-visual upsampling network. The audio prior helps to recover the elemental facial details and precise lip shapes, and a single high-resolution target identity image prior provides us with rich appearance details. Our approach is an end-to-end multi-stage framework. The first stage produces a coarse intermediate output video that can be then used to animate the single target identity image and generate realistic, accurate and high-quality outputs. Our approach is simple and performs exceedingly well (an 8× improvement in FID score) compared to previous super-resolution methods. We also extend our model to talking-face video compression, and show that we obtain a 3.5x improvement in terms of bits/pixel over the previous state-of-the-art. The results from our network are thoroughly analyzed through extensive ablation and comparative analysis and demonstration videos (in the paper and supplementary material).
ACM-MM

Lip-to-Speech Synthesis for Arbitrary Speakers in the Wild

Hegde, Sindhu, Prajwal, KR, Mukhopadhyay, Rudrabha, Namboodiri, Vinay P, and Jawahar, CV

In Proceedings of the 30th ACM International Conference on Multimedia (MM’22) 2022

Abs PDF Code Website

In this work, we address the problem of generating speech from silent lip videos for any speaker in the wild. In stark contrast to previous works in lip-to-speech synthesis, our work (i) is not restricted to a fixed number of speakers, (ii) does not explicitly impose constraints on the domain or the vocabulary and (iii) deals with videos that are recorded in the wild as opposed to within laboratory settings. The task presents a host of challenges with the key one being that many features of the desired target speech like voice, pitch and linguistic content cannot be entirely inferred from the silent face video. In order to handle these stochastic variations, we propose a new VAE-GAN architecture that learns to associate the lip and speech sequences amidst the variations. With the help of multiple powerful discriminators that guide the training process, our generator learns to synthesize speech sequences in any voice for the lip movements of any person. Extensive experiments on multiple datasets show that we outperform all baseline methods by a large margin. Further, our network can be fine-tuned on videos of specific identities to achieve a performance comparable to single-speaker models that are trained on 4x more data. We also conduct numerous ablation studies to analyze the effect of different modules of our architecture. A demo video in supplementary material demonstrates several qualitative results and comparisons.