Publications

2023

1. BMVC

   GestSync: Determining who is speaking without a talking head

   Hegde, Sindhu, and Zisserman, Andrew

   In British Machine Vision Conference (BMVC) 2023

   Abs PDF Code Website

   In this paper we introduce a new synchronisation task, Gesture-Sync: determining if a person’s gestures are correlated with their speech or not. In comparison to Lip-Sync, Gesture-Sync is far more challenging as there is a far looser relationship between the voice and body movement than there is between voice and lip motion. We introduce a dual-encoder model for this task, and compare a number of input representations including RGB frames, keypoint images, and keypoint vectors, assessing their performance and advantages. We show that the model can be trained using self-supervised learning alone, and evaluate its performance on the LRS3 dataset. Finally, we demonstrate applications of Gesture-Sync for audio-visual synchronisation, and in determining who is the speaker in a crowd, without seeing their faces.

2022

1. ACM-MM

   Extreme-scale Talking-Face Video Upsampling with Audio-Visual Priors

   Hegde, Sindhu, Mukhopadhyay, Rudrabha, Namboodiri, Vinay P, and Jawahar, CV

   In Proceedings of the 30th ACM International Conference on Multimedia (MM’22) 2022

   Abs PDF Code Website

   In this paper, we explore an interesting question of what can be obtained from an 8×8 pixel video sequence. Surprisingly, it turns out to be quite a lot. We show that when we process this 8x8 video with the right set of audio and image priors, we can obtain a full-length, 256x256 video. We achieve this 32x scaling of an extremely low-resolution input using our novel audio-visual upsampling network. The audio prior helps to recover the elemental facial details and precise lip shapes, and a single high-resolution target identity image prior provides us with rich appearance details. Our approach is an end-to-end multi-stage framework. The first stage produces a coarse intermediate output video that can be then used to animate the single target identity image and generate realistic, accurate and high-quality outputs. Our approach is simple and performs exceedingly well (an 8× improvement in FID score) compared to previous super-resolution methods. We also extend our model to talking-face video compression, and show that we obtain a 3.5x improvement in terms of bits/pixel over the previous state-of-the-art. The results from our network are thoroughly analyzed through extensive ablation and comparative analysis and demonstration videos (in the paper and supplementary material).

2. ACM-MM

   Lip-to-Speech Synthesis for Arbitrary Speakers in the Wild

   Hegde, Sindhu, Prajwal, KR, Mukhopadhyay, Rudrabha, Namboodiri, Vinay P, and Jawahar, CV

   In Proceedings of the 30th ACM International Conference on Multimedia (MM’22) 2022

   Abs PDF Code Website

   In this work, we address the problem of generating speech from silent lip videos for any speaker in the wild. In stark contrast to previous works in lip-to-speech synthesis, our work (i) is not restricted to a fixed number of speakers, (ii) does not explicitly impose constraints on the domain or the vocabulary and (iii) deals with videos that are recorded in the wild as opposed to within laboratory settings. The task presents a host of challenges with the key one being that many features of the desired target speech like voice, pitch and linguistic content cannot be entirely inferred from the silent face video. In order to handle these stochastic variations, we propose a new VAE-GAN architecture that learns to associate the lip and speech sequences amidst the variations. With the help of multiple powerful discriminators that guide the training process, our generator learns to synthesize speech sequences in any voice for the lip movements of any person. Extensive experiments on multiple datasets show that we outperform all baseline methods by a large margin. Further, our network can be fine-tuned on videos of specific identities to achieve a performance comparable to single-speaker models that are trained on 4x more data. We also conduct numerous ablation studies to analyze the effect of different modules of our architecture. A demo video in supplementary material demonstrates several qualitative results and comparisons.

2021

1. BMVC

   Audio-Visual Speech Super-Resolution

   Mukhopadhyay, Rudrabha,  Hegde, Sindhu, Namboodiri, Vinay P., and Jawahar, C.V.

   In British Machine Vision Conference (BMVC) 2021

   Abs PDF Website

   In this paper, we present an audio-visual model to perform speech super-resolution at large scale-factors (8x and 16x). Previous works attempted to solve this problem using only the audio modality as input, and thus were limited to low scale-factors of 2x and 4x. In contrast, we propose to incorporate both visual and auditory signals to super-resolve speech of sampling rates as low as 1kHz. In such challenging situations, the visual features assist in learning the content, and improves the quality of the generated speech. Further, we demonstrate the applicability of our approach to arbitrary speech signals where the visual stream is not accessible. Our “pseudo-visual network” precisely synthesizes the visual stream solely from the low-resolution speech input. Extensive experiments illustrate our method’s remarkable results and benefits over state-of-the-art audio-only speech super-resolution approaches.

2. WACV

   Visual Speech Enhancement Without a Real Visual Stream

   Hegde, Sindhu, Prajwal, K.R., Mukhopadhyay, Rudrabha, Namboodiri, Vinay P., and Jawahar, C.V.

   In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) 2021

   Abs PDF Code Website

   In this work, we re-think the task of speech enhancement in unconstrained real-world environments. Current state-of-the-art methods use only the audio stream and are limited in their performance in a wide range of real-world noises. Recent works using lip movements as additional cues improve the quality of generated speech over "audio-only" methods. But, these methods cannot be used for several applications where the visual stream is unreliable or completely absent. We propose a new paradigm for speech enhancement by exploiting recent breakthroughs in speech-driven lip synthesis. Using one such model as a teacher network, we train a robust student network to produce accurate lip movements that mask away the noise, thus acting as a "visual noise filter". The intelligibility of the speech enhanced by our pseudo-lip approach is comparable (< 3% difference) to the case of using real lips. This implies that we can exploit the advantages of using lip movements even in the absence of a real video stream. We rigorously evaluate our model using quantitative metrics as well as human evaluations. Additional ablation studies and a demo video on our website containing qualitative comparisons and results clearly illustrate the effectiveness of our approach.

3. INTERSPEECH

   Towards Automatic Speech to Sign Language Generation

   Parul, Kapoor, Mukhopadhyay, Rudrabha,  Hegde, Sindhu, Namboodiri, Vinay P., and Jawahar, C.V.

   In Interspeech 2021

   Abs PDF Code Website

   We aim to solve the highly challenging task of generating continuous sign language videos solely from speech segments for the first time. Recent efforts in this space have focused on gen- erating such videos from human-annotated text transcripts without considering other modalities. However, replacing speech with sign language proves to be a practical solution while communicating with people suffering from hearing loss. Therefore, we eliminate the need of using text as input and design techniques that work for more natural, continuous, freely uttered speech covering an extensive vocabulary. Since the current datasets are inadequate for generating sign language directly from speech, we collect and release the first Indian sign language dataset comprising speech-level annotations, text transcripts, and the corresponding sign-language videos. Next, we propose a multi-tasking transformer network trained to generate signer’s poses from speech segments. With speech-to-text as an auxiliary task and an additional cross-modal discriminator, our model learns to generate continuous sign pose sequences in an end-to-end manner. Extensive experiments and comparisons with other baselines demonstrate the effectiveness of our approach. We also conduct additional ablation studies to analyze the effect of different modules of our network.

4. Journal

   PIG-Net: Inception based Deep Learning Architecture for 3D Point Cloud Segmentation

   Hegde, Sindhu, and Gangisetty, Shankar

   Comput. Graph. 2021

   Abs PDF

   Point clouds, being the simple and compact representation of surface geometry of 3D objects, have gained increasing popularity with the evolution of deep learning networks for classification and segmentation tasks. Unlike human, teaching the machine to analyze the segments of an object is a challenging task and quite essential in various machine vision applications. In this paper, we address the problem of segmentation and labelling of the 3D point clouds by proposing a inception based deep network architecture called PIG-Net, that effectively characterizes the local and global geometric details of the point clouds. In PIG-Net, the local features are extracted from the transformed input points using the proposed inception layers and then aligned by feature transform. These local features are aggregated using the global average pooling layer to obtain the global features. Finally, feed the concatenated local and global features to the convolution layers for segmenting the 3D point clouds. We perform an exhaus- tive experimental analysis of the PIG-Net architecture on two state-of-the-art datasets, namely, ShapeNet [1] and PartNet [2]. We evaluate the effectiveness of our network by performing ablation study.

5. ACN

   Fake Review Detection Using Hybrid Ensemble Learning

   Hegde, Sindhu, Raj Rai, Raghu, Sunitha Hiremath, P. G., and Gangisetty, Shankar

   In Advances in Computing and Network Communications 2021

   Abs PDF

   Opinion spam on online restaurant review sites are a major problem as the reviews influence the users’ choice to visit or not to a restaurant. In this paper, we address the problem of detecting genuine and fake reviews in restaurant online reviews. We propose a fake review detection technique comprising data preprocessing, detection and ensemble learning that learns the reviews and their features to filter out the fake reviews. Initially, we preprocess to obtain the refined reviews and employ two independent classifiers using deep machine learning and feature-based machine learning techniques for detection. These classifiers tackle the problem in two aspects, i.e., the deep machine learning model learns the word distributions and the feature-based machine learning model extracts the relevant features from the reviews. Finally, a hybrid ensemble model from the two classifiers are built to detect the genuine and fake reviews. The experimental analysis of the proposed approach on Yelp datasets outperforms the existing state-of-the-art methods.

2019

1. BMVC

   An Evaluation of Feature Encoding Techniques for Non-Rigid and Rigid 3D Point Cloud Retrieval

   Hegde, Sindhu, and Gangisetty, Shankar

   In BMVC 2019

   Abs PDF

   In this paper, we address the 3D point cloud based retrieval problem for both non- rigid and rigid 3D data. As powerful computation resources and scanning devices have led to an exponential growth of 3D point cloud data, retrieving the relevant 3D objects from databases is a challenging task. The local descriptors provide only the abstract representations that do not enable the exploration of shape variability to solve the 3D object retrieval problem. Thus, it is not just the local descriptors but also the encoding of local signatures into global descriptors which is of crucial importance for enhancing the performance. To create a compact shape signature that constitutes the 3D object as a whole, various encoding techniques have been proposed in the literature. The most popular among them are bag-of-features [28], Fisher vector [20] and vector of locally aggregated descriptors [11]. However evaluating the different encoding techniques and analyzing the critical aspects to boost the performance of 3D point cloud retrieval is still an unsolved problem. We propose to provide an exhaustive evaluation of the different encoding techniques when combined with local feature descriptors for solving non-rigid and rigid point cloud retrieval task. We fix improved wave kernel signature and metric tensor & Christoffel symbols local descriptors specifically built for non-rigid and rigid data as given in [18] and [24] respectively. We also present a consistent comparative analysis of our method with the existing benchmarks, the results of which illustrate the robustness of the proposed approach on point cloud data.

2018

1. ICVGIP

   Evaluation of Point Cloud Categorization for Rigid and Non-Rigid 3D Objects

   Gangisetty, Shankar,  Hegde, Sindhu, Satyappanavar, Supriya, and Mudenagudi, Uma

   In Proceedings of the 11th Indian Conference on Computer Vision, Graphics and Image Processing (ICVGIP) 2018

   Abs PDF

   In this paper, we address the problem of 3D object categorization for point cloud data. With the availability of inexpensive scanning devices and powerful computational resources, there is a rapid growth of point cloud data. This necessitates efficient classification techniques which form the basis for analysis and processing of 3D point cloud data. In order to address the classification problem, we propose a 3D object categorization framework for both rigid and non-rigid objects. Initially, the proposed approach extracts the feature descriptors using improved wave kernel signature by approximating Laplace-Beltrami operator on point cloud data for non-rigid objects. For rigid objects, our approach uses the geometric features, namely, metric tensor and Christoffel symbols by modifying the geodesic distance computation. These feature descriptors are then represented using bag-of-features and improved Fisher vector encoding techniques. Finally, the support vector machine classifies the 3D objects into predefined set of classes. We also provide an exhaustive performance evaluation of the proposed 3D object categorization framework on state-of-the-art datasets, namely, SHREC’10, SHREC’11, SHREC’12, SHREC’15 and Princeton Shape Benchmark. The evaluation results reveal that the proposed approach outperforms the existing object categorization methods for both rigid and non-rigid 3D objects.

2. ICACCI

   Sentiment based Food Classification for Restaurant Business

   Hegde, Sindhu, Satyappanavar, Supriya, and Gangisetty, Shankar

   International Conference on Advances in Computing, Communications and Informatics (ICACCI) 2018

   Abs PDF

   In this paper, we address the problem of classifying the restaurant reviews into different meal courses based on the review sentiments. Based on our survey, we observe that there are no works which focus on classifying the food items into different meal courses with top n food items under each course using restaurant reviews. Most of the works in literature address the issues like predicting the ratings of the restaurant or restaurant business strategies by considering various attributes of the restaurant. Thus, in this paper, we propose a sentiment based food classification framework consisting of two tasks, namely, sentiment classification and four-course meal classification. In sentiment classification, we classify the reviews into positive and negative categories based on the sentiments of the reviews. In four-course meal classification, we categorize the reviews into four courses, namely, soups and salads, appetizers, main course and desserts. We list the top n food items liked by most of the customers in each of these courses. In order to select the suitable classification technique for addressing the identified problem, we analyze the learning curves based on the bias-variance trade-off. We observe that support vector machines (SVM) classification technique outperforms other techniques. The performance analysis of the proposed framework is carried out on the standard Yelp dataset. The top 5 food items obtained in each of the categories are soups and salads: salad, corn, coffee, cocktail, tea; appetizers: burger, pizza, bread, cheese, frie; main course: steak, meal, chicken, meat, beef; and desserts: doughnut, cream, cake, chocolate, flan.

2017

1. ICACCI

   Restaurant setup business analysis using yelp dataset

   Hegde, Sindhu, Satyappanavar, Supriya, and Gangisetty, Shankar

   International Conference on Advances in Computing, Communications and Informatics (ICACCI) 2017

   Abs PDF

   In this paper, we address the issues associated with setting-up of a new restaurant business. To strategize a new restaurant business, we propose a restaurant business framework which comprises of 3 most important tasks, namely, high frequency attributes, most crowded day and location of the restaurant. First, we identify the features/attributes of the restaurants in which the customers are most interested in and provide those facilities and services to increase the profit. Next, we identify the day of the week when the restaurants are heavily crowded so that the best recipes and offers are made available on those days. Finally, since location has a profound effect on the success of a restaurant business, we consider location to be the most important to know the nearby restaurants and their facilities before coming up with a new restaurant business. The performance analysis of the proposed framework was carried out on the standard Yelp dataset. Thus, we found credit card to be the most preferred attribute, the most crowded day to be Monday and Divey to be the most desired ambience among the customers. We also demonstrate how the new restaurant can be setup by identifying the nearest restaurants and the services.