1. κΉŠμ€ κ°•ν™”ν•™μŠ΅(RL)의 λ°œμ „μ€ 인간 μ—°κ΅¬μžλ“€μ΄ μˆ˜λ™μœΌλ‘œ μ„€κ³„ν•œ μ•Œκ³ λ¦¬μ¦˜μ— κΈ°λ°˜ν•˜μ˜€μœΌλ‚˜, μ΅œκ·Όμ—λŠ” 메타 ν•™μŠ΅ μ—…λ°μ΄νŠΈ κ·œμΉ™μ„ 톡해 λ‹€μ–‘ν•œ RL μž‘μ—…μ—μ„œ 잘 μˆ˜ν–‰ν•  수 μžˆλŠ” μ•Œκ³ λ¦¬μ¦˜μ„ λ°œκ²¬ν•˜λŠ” 것이 κ°€λŠ₯ν•˜λ‹€λŠ” 것이 λ°ν˜€μ‘Œλ‹€.


2. Unsupervised Environment Design (UED)μ—μ„œ μ˜κ°μ„ λ°›μ•„, μš°λ¦¬λŠ” 메타 ν•™μŠ΅λœ μ΅œμ ν™”κΈ°μ˜ ν›„νšŒλ₯Ό κ·ΉλŒ€ν™”ν•˜κΈ° μœ„ν•΄ μžλ™μœΌλ‘œ ꡐ윑 과정을 μƒμ„±ν•˜λŠ” μƒˆλ‘œμš΄ 접근법과 "μ•Œκ³ λ¦¬μ¦˜μ  ν›„νšŒ"λΌλŠ” μƒˆλ‘œμš΄ ν›„νšŒ 근사λ₯Ό μ œμ•ˆν•œλ‹€.

3. μ‹€ν—˜μ„ 톡해, 우리의 방법 GROOVEλŠ” LPG보닀 μš°μ›”ν•œ μΌλ°˜ν™”λ₯Ό λ‹¬μ„±ν•˜λ©°, 이 접근법은 μ‹€μ œ μ„Έκ³„μ˜ λ‹€μ–‘ν•œ ν™˜κ²½μ„ ν•΄κ²°ν•  수 μžˆλŠ” μ§„μ •ν•œ 일반 RL μ•Œκ³ λ¦¬μ¦˜μ˜ λ°œκ²¬μ„ ν–₯ν•œ ν•œ 걸음이라고 λ―ΏλŠ”λ‹€.


https://arxiv.org/abs/2310.02782

Discovering General Reinforcement Learning Algorithms with Adversarial Environment DesignThe past decade has seen vast progress in deep reinforcement learning (RL) on the back of algorithms manually designed by human researchers. Recently, it has been shown that it is possible to meta-learn update rules, with the hope of discovering algorithms that can perform well on a wide range of RL tasks. Despite impressive initial results from algorithms such as Learned Policy Gradient (LPG), there remains a generalization gap when these algorithms are applied to unseen environments. In this work, we examine how characteristics of the meta-training distribution impact the generalization performance of these algorithms. Motivated by this analysis and building on ideas from Unsupervised Environment Design (UED), we propose a novel approach for automatically generating curricula to maximize the regret of a meta-learned optimizer, in addition to a novel approximation of regret, which we name algorithmic regret (AR). The result is our method, General RL Optimizers Obtained Via Environment Design (GROOVE). In a series of experiments, we show that GROOVE achieves superior generalization to LPG, and evaluate AR against baseline metrics from UED, identifying it as a critical component of environment design in this setting. We believe this approach is a step towards the discovery of truly general RL algorithms, capable of solving a wide range of real-world environments.arxiv.org