本章将介绍推动RLHF（人类反馈强化学习）领域发展至今的核心论文和项目。
这不是一份关于RLHF及其相关领域的全面综述，而是一个起点，带你梳理这一路走来的关键节点。
内容有意聚焦于促成ChatGPT诞生的近年工作。
在RL领域，关于偏好学习还有大量更深入的研究 [1]。
如果你需要更详尽的文献列表，建议查阅专门的综述论文 [2],
[3]。
近年来，随着深度强化学习（Deep
RL）的发展，这一领域逐渐被大众熟知，并扩展为各大科技公司探索大语言模型（LLM）应用的主流方向之一。
不过，今天许多RLHF核心技术其实都可以追溯到早期关于“基于偏好强化学习”的文献。
人类反馈强化学习（RLHF），早期也常被称为“基于人类偏好的强化学习”，很快被AI实验室采纳，成为大语言模型扩展能力的重要方法。
大量相关工作始于2019年的GPT-2与2020年的GPT-3之间。 2019年最早的代表作
Fine-Tuning Language Models from Human Preferences
与现代RLHF研究有诸多相似之处 [10]，如奖励模型、KL距离、反馈流程图等——只是当时的评测任务和模型能力有所不同。
此后，RLHF被广泛应用于多种任务。 当时最受关注的应用领域包括： -
通用文本摘要 [11] - 递归式书籍摘要 [12] -
指令遵循（InstructGPT）[13] - 浏览器辅助问答（WebGPT）[14] -
答案引用支持（GopherCite）[15] - 通用对话（Sparrow）[16]
如今，RLHF正逐步发展为更广义的“偏好微调”（Preference
Fine-Tuning，PreFT）领域，涵盖了许多新兴应用，比如： -
针对中间推理步骤的过程奖励（process reward）[30] - 受直接偏好优化（DPO, Direct
Preference Optimization）启发的直接对齐算法 [31] - 基于代码或数学题执行反馈的学习
[32],
[33] -
以及受OpenAI o1启发的在线推理方法 [34]
[1]
C. Wirth, R. Akrour, G. Neumann, 和 J.
Fürnkranz, 《A survey of preference-based reinforcement learning
methods》, Journal of Machine Learning Research, 卷
18, 期 136, 页 1～46, 2017.
[2]
T. Kaufmann, P. Weng, V. Bengs, 和 E.
Hüllermeier, 《A survey of reinforcement learning from human
feedback》, arXiv preprint arXiv:2312.14925,
2023.
[3]
S. Casper
等, 《Open problems and fundamental limitations
of reinforcement learning from human feedback》, arXiv
preprint arXiv:2307.15217, 2023.
[4]
W. B. Knox 和 P. Stone, 《Tamer:
Training an agent manually via evaluative reinforcement》, 收入
2008 7th IEEE international conference on development and
learning, IEEE, 2008, 页 292～297.
[5]
J. MacGlashan 等,
《Interactive learning from policy-dependent human
feedback》, 收入 International conference on machine
learning, PMLR, 2017, 页 2285～2294.
[6]
P. F. Christiano, J. Leike, T. Brown, M.
Martic, S. Legg, 和 D. Amodei, 《Deep reinforcement learning
from human preferences》, Advances in neural information
processing systems, 卷 30, 2017.
[7]
B. Ibarz, J. Leike, T. Pohlen, G. Irving, S.
Legg, 和 D. Amodei, 《Reward learning from human preferences and
demonstrations in atari》, Advances in neural information
processing systems, 卷 31, 2018.
[8]
G. Warnell, N. Waytowich, V. Lawhern, 和 P.
Stone, 《Deep tamer: Interactive agent shaping in
high-dimensional state spaces》, 收入 Proceedings of the
AAAI conference on artificial intelligence, 2018.
[9]
J. Leike, D. Krueger, T. Everitt, M. Martic,
V. Maini, 和 S. Legg, 《Scalable agent alignment via reward
modeling: a research direction》, arXiv preprint
arXiv:1811.07871, 2018.
[10]
D. M. Ziegler 等,
《Fine-tuning language models from human preferences》,
arXiv preprint arXiv:1909.08593, 2019.
[11]
N. Stiennon 等, 《Learning to
summarize with human feedback》, Advances in Neural
Information Processing Systems, 卷 33, 页 3008～3021, 2020.
[12]
J. Wu 等, 《Recursively
summarizing books with human feedback》, arXiv preprint
arXiv:2109.10862, 2021.
[13]
L. Ouyang
等, 《Training language models to follow
instructions with human feedback》, Advances in neural
information processing systems, 卷 35, 页 27730～27744,
2022.
[14]
R. Nakano
等, 《Webgpt: Browser-assisted
question-answering with human feedback》, arXiv preprint
arXiv:2112.09332, 2021.
[15]
J. Menick
等, 《Teaching language models to support
answers with verified quotes》, arXiv preprint
arXiv:2203.11147, 2022.
[16]
A. Glaese
等, 《Improving alignment of dialogue agents via
targeted human judgements》, arXiv preprint
arXiv:2209.14375, 2022.
[17]
L. Gao, J. Schulman, 和 J. Hilton,
《Scaling laws for reward model overoptimization》, 收入
International Conference on Machine Learning, PMLR, 2023, 页
10835～10866.
[18]
A. Askell
等, 《A general language assistant as a
laboratory for alignment》, arXiv preprint
arXiv:2112.00861, 2021.
[19]
D. Ganguli
等, 《Red teaming language models to reduce
harms: Methods, scaling behaviors, and lessons learned》,
arXiv preprint arXiv:2209.07858, 2022.
[20]
Y. Bai
等, 《Training a helpful and harmless assistant
with reinforcement learning from human feedback》, arXiv
preprint arXiv:2204.05862, 2022.
[21]
R. Ramamurthy 等, 《Is
reinforcement learning (not) for natural language processing:
Benchmarks, baselines, and building blocks for natural language policy
optimization》, arXiv preprint arXiv:2210.01241,
2022.
[22]
A. Havrilla
等,
《trlX: A Framework for Large Scale Reinforcement
Learning from Human Feedback》, 收入
Proceedings of the
2023 Conference on Empirical Methods in Natural Language
Processing, Singapore: Association for Computational Linguistics,
12月 2023, 页 8578～8595. doi:
10.18653/v1/2023.emnlp-main.530.
[25]
Y. Bai
等, 《Constitutional ai: Harmlessness from ai
feedback》, arXiv preprint arXiv:2212.08073,
2022.
[26]
H. Touvron
等, 《Llama 2: Open foundation and fine-tuned
chat models》, arXiv preprint arXiv:2307.09288,
2023.
[27]
A. Dubey
等, 《The llama 3 herd of models》,
arXiv preprint arXiv:2407.21783, 2024.
[28]
B. Adler
等, 《Nemotron-4 340B Technical Report》,
arXiv preprint arXiv:2406.11704, 2024.
[29]
N. Lambert
等, 《T\(\backslash\)" ULU 3: Pushing Frontiers in
Open Language Model Post-Training》, arXiv preprint
arXiv:2411.15124, 2024.
[30]
H. Lightman 等, 《Let’s verify
step by step》, arXiv preprint arXiv:2305.20050,
2023.
[31]
R. Rafailov, A. Sharma, E. Mitchell, C. D.
Manning, S. Ermon, 和 C. Finn, 《Direct preference optimization:
Your language model is secretly a reward model》, Advances
in Neural Information Processing Systems, 卷 36, 2024.
[32]
A. Kumar
等, 《Training language models to self-correct
via reinforcement learning》, arXiv preprint
arXiv:2409.12917, 2024.
[33]
A. Singh
等, 《Beyond human data: Scaling self-training
for problem-solving with language models》, arXiv preprint
arXiv:2312.06585, 2023.