Data Exploration in a Camera-first World: Query and Result Challenges

Abstract: The pervasive availability of cameras in smartphones, vehicles, drones and more has triggered a new “camera-first” data revolution across industries. When combined with rapid advances in computer vision and machine learning, this video data deluge presents several data exploration challenges. Here, we will talk about two complementary challenges for end-users: query specification and result consumption.

When searching large video collections, the first challenge is that the user is often unaware of the contents of the video, its structure, and the exact terminology to use in the user query, putting them at a loss for where to begin specifying the query. Here, we present methods to guide the user through the query construction process by building on vision language models and search query interfaces.

Once users have executed a search, they are faced with a new challenge of result consumption. Presenting query results as a list of links poses an impedance mismatch: they are cumbersome to skim through and are in a different modality compared to the source data. However, processing large video collections within interactive response times has performance implications. We present V2V, a system to efficiently synthesize video results for video queries. V2V returns a fully-edited video, allowing the user to consume results in the same modality as the source videos, resulting in a fluid, user-centric video exploration experience.

Bio: Arnab’s work focuses on bridging data infrastructure with human interaction, spanning areas of database systems, human factors, and next-generation interfaces. Arnab is a recipient of the US National Science Foundation’s CAREER Award, IEEE’s TCDE Early Career Award for his contributions towards user-focused data interaction, The Ohio State University’s Alumni Award for Distinguished Teaching, and the University’s Early Career Innovator of the Year Award.

Over the years, Arnab has served as Program Committee member and Associate Editor for several database systems journals and conferences including SIGMOD, VLDB, ICDE, and HILDA. Most recently, Arnab served as Vice President of Data Science at Azuga Inc. (a Bridgestone company) after the acquisition of his connected vehicles analytics startup, Mobikit. https://arnab.org/