More from Less: Learning with Limited Annotated Data in Vision and Language

Abstract: Despite the impressive results of deep learning models, modern large-scale systems are required to be trained using massive amounts of manually annotated or freely available data on the Internet. But this “data in the wild” is insufficient to learn specific structural patterns of the world, and existing large-scale models still fail on common sense tasks requiring compositional inference. – This talk will focus on answering three fundamental questions: (a) How can we create systems that can learn with limited annotated data and adapt to new tasks and novel criteria? (b) How can we create systems able to encode real-world concepts with granularity in a robust manner? (c) Is it possible to create such a system with alternative data, complying with privacy protection principles and avoiding cultural bias? – Given my work’s intersection with Computer Vision and Natural Language Processing, my aim is to analyze and apply Machine Learning algorithms to understand how images and text can interact and model complex patterns, reinforcing compositional reasoning without forgetting prior knowledge. Finally, I will conclude with my future plans to continue exploring hyper-realistic synthetic data generation techniques and the expressiveness of generative models to train multimodal systems able to perform well in real-world scenarios, with applications including visual-question answering, cross-modal retrieval, zero-shot classification, and task planning.

Bio: Paola Cascante-Bonilla is a Ph.D. Candidate in Computer Science at Rice University, working on Computer Vision, Natural Language Processing, and Machine Learning. She has been focusing on multi-modal learning, few-shot learning, semi-supervised learning, representation learning, and synthetic data generation for compositionality and privacy protection. Her work has been published in machine learning, vision, and language conferences (CVPR, ICCV, AAAI, NeurIPS, BMVC, NAACL). She has previously interned at the Mitsubishi Electric Research Laboratories (MERL) and twice at the MIT-IBM Watson AI Lab. She is the recipient of the Ken Kennedy Institute SLB Graduate Fellowship (2022/23), and has been recently selected as a Future Faculty Fellow by Rice’s George R. Brown School of Engineering (2023) and as a Rising Star in EECS (2023).

Webpage: https://paolacascante.com/