CompSci 295 - Causal Inference for Reinforcement Learning (Winter 2025)

Instructor: Rina Dechter Days,Time: M/W, 11:00 am - 12:20 pm (PT) Classoom: DBH 1300 Zoom Link: https://uci.zoom.us/j/94542138848 Office hours: Upon Request The class will cover topics in Causal reasoning focusing primarily on Reinforcement learning. The class will run as a seminar. I will give the first few introductory classes. Students will present present papers from the literature or chapters in books and complete a final project. There may also be some homework assignments. The course is intended primarily for PhD students in the area of AI and Machine Learning, but sufficient preparation in AI and Machine learning is expected and required (at least one of CS 171, 271, 273 or 276). Relevant sources (books or classes) On Graphical Models: [Dechter-book] Reasoning With Probabilistic and Deterministic Graphical Models: Exact Algorithms Rina Dechter [Dechter-class] Course: Causal and Probabilistic Reasoning with Graphical Models Rina Dechter On Causality: [Primer] Causal Inference in Statistics: A Primer Judea Pearl, Madelyn Glymour, Nicholas P. Jewell [CMRI] Causality: Models, Reasoning, and Inference Judea Pearl [Why] The Book of Why Judea Pearl, Dana Mackenzie [PH] On Pearl's Hierarchy and the Foundations of Causal Inference Elias Bareinboim, Juan D. Correa, Duligur Ibeling, Thomas Icard [CAI] Causal Artificial Intelligence: A Roadmap for Building Causally Intelligent Systems (preliminary draft) Elias Bareinboim On Reinforcement Learning: Reinforcement Learning: An Introduction Richard S. Sutton and Andrew G. Barto UCL Course on Reinforcement Learning David Silver Reinforcement Learning: Wikipedia Tutorials: Videos: Causal Reinforcement Learning Elias Bareinboim ICML 2020 tutorial series connecting causal inference and reinforcement learning, and describing specific Causal RL tasks, with list of related background papers. Towards Causal RL Part of an NSF seminar series, Elias Bareinboim's tutorial introducing Causal Reinforcement Learning and motivating its potential. Causal Fairness Analysis Part of ICML 2020, Elias Bareinboim and Drago Plecko's presentation on using causality to address fairness issues in AI decision making. Papers: Why Online Reinforcement Learning is Causal Oliver Schulte, Pascal Poupart Causal Reinforcement Learning: A Survey Zhihong Deng, Jing Jiang, Guodong Long, Chengqi Zhang Causal Machine Learning: A Survey and Open Problems Jean Kaddour, Aengus Lynch, Qi Liu, Matt J. Kusner, Ricardo Silva Causal RL An Introduction to Causal Reinforcement Learning Elias Bareinboim, Junzhe Zhang, Sanghack Lee Elias Bareinboim Papers Related to Task 1: Causal Online To Offline Learning Transfer Learning in Multi-Armed Bandits: A Causal Approach Junzhe Zhang, Elias Bareinboim Near-Optimal Reinforcement Learning in Dynamic Treatment Regimes Junzhe Zhang, Elias Bareinboim Designing Optimal Dynamic Treatment Regimes: A Causal Reinforcement Learning Approach Junzhe Zhang, Elias Bareinboim Off-policy Policy Evaluation For Sequential Decisions Under Unobserved Confounding Hongseok Namkoong, Ramtin Keramati, Steve Yadlowsky, Emma Brunskill Elias Bareinboim Papers Related to Task 2: When and Where to Intervene? (refining the policy space) Structural Causal Bandits: Where to Intervene? Sanghack Lee, Elias Bareinboim Structural Causal Bandits with Non-manipulable Variables Sanghack Lee, Elias Bareinboim Characterizing Optimal Mixed Policies: Where to Intervene and What to Observe Sanghack Lee, Elias Bareinboim Online Reinforcement Learning for Mixed Policy Scopes Junzhe Zhang, Elias Bareinboim Elias Bareinboim Papers Related to Task 3: Counterfactual Decision-Making (changing optimization function based on intentionality, free will, and autonomy) Bandits with Unobserved Confounders: A Causal Approach Elias Bareinboim, Andrew Forney Counterfactual Data-Fusion for Online Reinforcement Learners Andrew Forney, Judea Pearl, Elias Bareinboim Counterfactual Randomization: Rescuing Experimental Studies from Obscured Confounding Andrew Forney, Elias Bareinboim Can Humans Be out of the Loop? Junzhe Zhang, Elias Bareinboim Elias Bareinboim Papers Related to Task 7: Causal Curriculum Learning Causally Aligned Curriculum Learning Mingxuan Li, Junzhe Zhang, Elias Bareinboim Additional Papers: AdaRL: What, Where, and How to Adapt in Transfer Reinforcement Learning Biwei Huang, Fan Feng, Chaochao Lu, Sara Magliacane, Kun Zhang Towards Generalizable Reinforcement Learning via Causality-Guided Self-Adaptive Representations Yupei Yang, Biwei Huang, Fan Feng, Xinyue Wang, Shikui Tu, Lei Xu Learning World Models with Identifiable Factorization Yu-Ren Liu, Biwei Huang, Zhengmao Zhu, Honglong Tian, Mingming Gong, Yang Yu, Kun Zhang False Correlation Reduction for Offline Reinforcement Learning Zhihong Deng, Zuyue Fu, Lingxiao Wang, Zhuoran Yang, Chenjia Bai, Tianyi Zhou, Zhaoran Wang, Jing Jiang Causal Reinforcement Learning using Observational and Interventional Data Maxime Gasse, Damien Grasset, Guillaume Gaudron, Pierre-Yves Oudeyer Causal Imitation Learning with Unobserved Confounders Junzhe Zhang, Daniel Kumor, Elias Bareinboim Provably Efficient Causal Reinforcement Learning with Confounded Observational Data Lingxiao Wang, Zhuoran Yang, Zhaoran Wang CausalWorld: A Robotic Manipulation Benchmark for Causal Structure and Transfer Learning Ossama Ahmed, Frederik Träuble, Anirudh Goyal, Alexander Neitz, Yoshua Bengio, Bernhard Schölkopf, Manuel Wüthrich, Stefan Bauer Causality-driven Hierarchical Structure Discovery for Reinforcement Learning Shaohui Peng, Xing Hu, Rui Zhang, Ke Tang, Jiaming Guo, Qi Yi, Ruizhi Chen, Xishan Zhang, Zidong Du, Ling Li, Qi Guo, Yunji Chen Papers and Lectures on Fairness Course on Causal Fairness Analysis and related papers Drago Plecko From Pearl's group: Personal decision making, Robustness Personalized decision making – A conceptual introduction Scott Mueller, Judea Pearl Unit Selection Based on Counterfactual Logic Ang Li, Judea Pearl Causal inference and the data-fusion problem Elias Bareinboim, Judea Pearl External Validity: From Do-Calculus to Transportability Across Populations Judea Pearl, Elias Bareinboim Tools for RL: Miscellaneous Tools for RL Course Project Each student will also be engaged in a project based on papers from recent literature. The project will involve learning about and preseting an assigned paper/literature in class and a final project report. Project Description Project Sign-up Sheet Schedule

Week	Date	Topic	Readings and Links
Week 1	M 01/06, W 01/08	Tutorial on Causal Probabilistic Graphical Models	Assignments - HW 1 Slides - Slides 1 - Slides 2 Related Resources - [Primer] - [Dechter-book] - [Dechter-class]
Week 2	M 01/13, W 01/15	Introduction to Reinforcement Learning	Slides - Slides 3 - Slides 4
Week 3	M 01/20 , W 01/22	Presentations 1. Jiapeng Zhao: "Introduction to Causal Reinforcement Learning" [pdf]     (Discussant: Rina Dechter)	Slides - P1 Slides
Week 4	M 01/27, W 01/29	Presentations 1.(cont'd) Jiapeng Zhao: "Introduction to Causal Reinforcement Learning" [pdf] [video (part II of presentation)]     (Discussant: Rina Dechter) 2. Omar Samiullah: "Transfer Learning in Multi-Armed Bandits: A Causal Approach" [pdf] [video (full)]     (Discussant: Annie Raichev) 3. Jared Macshane: "Online Reinforcement Learning for Mixed Policy Scopes" [pdf] [video]     (Discussant: Bryan Vela)	Assignments - HW 2 Slides - P1 Slides - P2 Slides - P3 Slides
Week 5	M 02/03, W 02/05, Th 02/06	Presentations (M 02/03) 2.(cont'd) Omar Samiullah: "Transfer Learning in Multi-Armed Bandits: A Causal Approach" [pdf] [video (full)]     (Discussant: Annie Raichev) 4. David Lee: "Designing Optimal Dynamic Treatment Regimes: A Causal Reinforcement Learning Approach" [pdf] [video]     (Discussant: Shaoyuan Xie)	Assignments - HW 3 Slides - P2 Slides - P4 Slides Agenda for Workshop on Causal Reasoning at UCI
Week 6	M 02/10, W 02/12	Presentations 5. Michael Mulder: "Causal Imitation Learning Via Inverse Reinforcement Learning" [pdf] and "Causal Imitation Learning with Unobserved Confounders" [pdf] [video]     (Discussant: Omar Samiullah) 6. Annie Raichev: "Causal Reinforcement Learning using Observational and Interventional Data" [pdf] [video]     (Discussant: Brandon Vela)	Assignments - HW 4 Slides - P5 Slides - P6 Slides
Week 7	M 02/17 , W 02/19	Presentations 7. Jay Yoo: "Personalized decision making - A conceptual introduction" [pdf] and "Unit Selection Based on Counterfactual Logic" [pdf] [video]     (Discussant: Jared Macshane)	Assignments - HW 5 Slides - P7 Slides
Week 8	M 02/24, W 02/26	Presentations 8. Bryan Vela: "Fairness in Decision-Making - The Causal Explanation Formula" [pdf] [video]     (Discussant: David Lee) 9. Brandon Vela: "Can Humans Be out of the Loop?" [pdf] [video]     (Discussant: Michael Mulder)	Assignments - See HW 5 from week 8 - HW 6 Slides - P8 Slides - P9 Slides
Week 9	M 03/03, W 03/05	Presentation (Wednesday [in-person]) 10. Shaoyuan Xie: "Towards Generalizable Reinforcement Learning via Causality-Guided Self-Adaptive Representations" [pdf] [video]     (Discussant: Jay Yoo)	Assignments - HW 7 Slides - P10 Slides
Week 10	M 03/10, W 03/12	Presentations 11. Jiapeng Zhao: "The Causal-Neural Connection: Expressiveness, Learnability, and Inference" [pdf] [video]     (Discussant: Class) 12. (Guest Speaker) Mingxuan Li: "Automatic Reward Shaping from Confounded Offline Data" [pdf]     (Discussant: Class)	Assignments - See HW 7 from Week 9 - Final projects due Wednesday, March 19th Slides - P11 Slides - P12 Slides
Finals Week	F 03/21, 8:00-10:00am