Let's Verify Step by StepIn recent years, large language models have greatly improved in their ability to perform complex multi-step reasoning. However, even state-of-the-art models still regularly produce logical mistakes. To train more reliable models, we can turn either to outcome supervision, which provides feedback for a final result, or process supervision, which provides feedback for each intermediate reasoning step. Given the importance of training reliable models, and given the high cost of human feedback, it is important to carefully compare the both methods. Recent work has already begun this comparison, but many questions still remain. We conduct our own investigation, finding that process supervision significantly outperforms outcome supervision for training models to solve problems from the challenging MATH dataset. Our process-supervised model solves 78% of problems from a representative subset of the MATH test set. Additionally, we show that active learning significantly improves the efficacy of process supervision. To support related research, we also release PRM800K, the complete dataset of 800,000 step-level human feedback labels used to train our best reward model.arxiv.org

올해 5월 논문
일리야 참여함
reasoning과 관련됨
MATH에서 당시 sota 78% 달성
RLHF의 reward model 학습위한 수학 step-level 데이터셋

알다싶이 RLHF는 gpt를 policy network로 취급하고 ppo 써서 사람이 선호하게 출력하도록 파인튜닝한건데, Q스타는 이름부터 value based인 q러닝이라 gpt가 수학적 추론하는 과정에서 가능한 state랑 action value를 step level로 reward준걸 ㅈㄴ발전시킨거 아닐까? 싶음
근데 이건 학계에서 이미 여러번 시도해봤던 것들이고 (결과는 걍 별로였음)
A스타 알고리즘써서 추론 경로 탐색할거란 추측도 있던데 암튼 무슨모델이 나올지 궁금함

- dc official App