“Risk and robustness in RL: Nothing ventured, nothing gained”

In this talk I will start from giving a broad overview of my research, focusing on the essential elements needed for scaling reinforcement learning  to real-world problems. I will present a scheme called "extended intelligence" that concerns the design of systems that participate as responsible, aware and robust elements of more complex systems. I will then deep dive into  the question of how to create control policies from existing historical data and how to  sample trajectories so that future control policies would have less uncertain return. This question has been central in reinforcement learning in the last decade if not more, and involves methods from statistics, optimization, and control theory. We will focus on one the possible remedies to uncertainty in sequential decision problems: using risk measures such as the conditional value-at-risk as the objective to be optimized rather than the ubiquitous expected reward. We consider the complexity and efficiency of evaluating and optimizing risk measures. Our main theme is that considering risk is essential to obtain resilience to model uncertainty and  model mismatch. We then turn our attention to online approaches that adapt on-the-fly to the level of uncertainty of a given trajectory, thus achieving robustness without being overly conservative. If time permits, I will shortly discuss a couple of real-world applications my group has been working: one in energy management and one in healthcare.

Shie Mannor graduated from the Technion – Israel Institute of Technology with a BSc in electrical engineering and BA in mathematics (both summa cum laude) in 1996. After that, he spent almost four years as an intelligence officer with the Israeli Defense Forces. He earned his Ph.D. in electrical engineering from the Technion in 2002, under the supervision of Nahum Shimkin. He was a Fulbright postdoctoral associate with LIDS (MIT), working with John Tsitsiklis for two years. He was in the Department of Electrical and Computer Engineering at McGill University from July 2004 to August 2010, where he held a Canada Research Chair in Machine Learning from 2005 to 2009. He has been a professor with the Department of Electrical Engineering at the Technion since 2008.