“Risk
and robustness in RL: Nothing ventured, nothing gained”
In this talk I will start from
giving a broad overview of my research, focusing on the essential elements
needed for scaling reinforcement learning to real-world problems. I
will present a scheme called "extended intelligence" that concerns
the design of systems that participate as responsible, aware and robust
elements of more complex systems. I will then deep dive into the
question of how to create control policies from existing historical data and
how to sample trajectories so that future control policies would
have less uncertain return. This question has been central in reinforcement
learning in the last decade if not more, and involves methods from statistics,
optimization, and control theory. We will focus on one the possible remedies to
uncertainty in sequential decision problems: using risk measures such as the
conditional value-at-risk as the objective to be optimized rather than the
ubiquitous expected reward. We consider the complexity and efficiency of
evaluating and optimizing risk measures. Our main theme is that considering
risk is essential to obtain resilience to model uncertainty
and model mismatch. We then turn our attention to online approaches
that adapt on-the-fly to the level of uncertainty of a given trajectory, thus
achieving robustness without being overly conservative. If time permits, I will
shortly discuss a couple of real-world applications my group has been working:
one in energy management and one in healthcare.
Shie Mannor graduated from the
Technion – Israel Institute of Technology with a BSc in electrical engineering
and BA in mathematics (both summa cum laude) in 1996. After that, he spent
almost four years as an intelligence officer with the Israeli Defense Forces.
He earned his Ph.D. in electrical engineering from the Technion in 2002, under
the supervision of Nahum Shimkin. He was a Fulbright postdoctoral associate
with LIDS (MIT), working with John Tsitsiklis for two years. He was in the
Department of Electrical and Computer Engineering at McGill University from July
2004 to August 2010, where he held a Canada Research Chair in Machine Learning
from 2005 to 2009. He has been a professor with the Department of Electrical
Engineering at the Technion since 2008.