Buggy Rides: Uncovering Software Defects in Autonomous Driving Systems

Abstract: An autonomous vehicle (AV), often called a self-driving car, is operated by an autonomous driving system (ADS). AVs are becoming a pervasive and ubiquitous part of our daily lives. AVs are already driving public passengers in some major cities around the world (e.g., Alphabet’s Waymo in San Francisco and Los Angeles; Baidu Apollo Go in Wuhan, China; and Tier-IV in Tokyo, Japan), and more than 50 corporations and governments are actively working on or using AVs, including large companies such as Ford, Intel, ARM, Alibaba, and Tesla. To advance ADS testing, my research group has been studying bugs in ADSes and producing software testing techniques that (1) bridge the gap between simply finding traffic violations to guaranteeing that such discovered driving violations, especially collisions, stem from bugs in an ADS; (2) overcoming or reducing constraints preventing efficient and more diverse bug-revealing scenarios in ADS testing; and (3) generating tests that can efficiently and effectively evaluate the exponentially large configuration space of ADSes. Our software testing techniques find up to 8 times more unique bug-revealing violations than state-of-the-art approaches for around 10 different bug types in the same amount of time (i.e., over 1,500 hours of testing in simulation). Across all our solutions and research artifacts, my research group and I have contributed 10 confirmed bugs and 4 accepted fixes to Apollo and Autoware, two of the most widely-used, industry- or near-industry-grade, open-source ADSes—which collectively have been deployed to drive public passengers for robotaxis in major cities in China and Japan. I will end my talk by touching upon my research group’s ongoing and future research directions on software testing and analysis of ADSes involving responsibility and avoidability of driving violations to non-determinism (a.k.a flakiness).

Bio: Joshua Garcia is an assistant professor at the Informatics Department of the Donald Bren School of Information and Computer Sciences at the University of California, Irvine. His research interests are in software engineering, especially automated software analysis using dynamic program analysis, static program analysis, and artificial intelligence (including machine learning). He leverages empirical software engineering and qualitative methods to inform his automated solutions. Using such methods, he focuses on three problem domains: (1) software testing and analysis of autonomous driving systems, (2) software vulnerability management, and (3) the decay of software architectures. He received three degrees from the University of Southern California: a B.S. in computer engineering and computer science, an M.S. in computer science, and a Ph.D. in computer science. His research tools and datasets have been used by dozens of researchers, practitioners, agencies, and companies around the world—including universities in Argentina, Australia, Brazil, Canada, China, Europe, and the United States, and by companies and government agencies such as Boeing, Bosch, Google, HORIBA MIRA, IBM, Microsoft, Northrop Grumman, the FBI, the Department of Homeland Security, and NASA. His efforts have been funded by various sources, including the National Science Foundation; DARPA; and Nippon Telegraph and Telephone, Japan’s largest telecommunications company. He is also passionate about diversity, equity, inclusion, and accessibility (DEIA), e.g., serving as co-chair for DEIA at the 45th International Conference on Software Engineering (ICSE 2023), the top conference in software engineering research.