ORIE Colloquium: Shie Mannor (Technion), Feb. 23, 2021
From Henry Lam
“Risk and robustness in RL: Nothing ventured, nothing gained”
In this talk I will start from giving a broad overview of my research, focusing on the essential elements needed for scaling reinforcement learning to real-world problems. I will present a scheme called "extended intelligence" that concerns the design of systems that participate as responsible, aware and robust elements of more complex systems. I will then deep dive into the question of how to create control policies from existing historical data and how to sample trajectories so that future control policies would have less uncertain return. This question has been central in reinforcement learning in the last decade if not more, and involves methods from statistics, optimization, and control theory. We will focus on one the possible remedies to uncertainty in sequential decision problems: using risk measures such as the conditional value-at-risk as the objective to be optimized rather than the ubiquitous expected reward. We consider the complexity and efficiency of evaluating and optimizing risk measures. Our main theme is that considering risk is essential to obtain resilience to model uncertainty and model mismatch. We then turn our attention to online approaches that adapt on-the-fly to the level of uncertainty of a given trajectory, thus achieving robustness without being overly conservative. If time permits, I will shortly discuss a couple of real-world applications my group has been working: one in energy management and one in healthcare.
Shie Mannor graduated from the
Technion – Israel Institute of Technology with a BSc in electrical engineering
and BA in mathematics (both summa cum laude) in 1996. After that, he spent
almost four years as an intelligence officer with the Israeli Defense Forces.
He earned his Ph.D. in electrical engineering from the Technion in 2002, under
the supervision of Nahum Shimkin. He was a Fulbright postdoctoral associate
with LIDS (MIT), working with John Tsitsiklis for two years. He was in the
Department of Electrical and Computer Engineering at McGill University from July
2004 to August 2010, where he held a Canada Research Chair in Machine Learning
from 2005 to 2009. He has been a professor with the Department of Electrical
Engineering at the Technion since 2008.