Looks like the publisher may have taken this series offline or changed its URL. Please contact support if you believe it should be working, the feed URL is invalid, or you have any other concerns about it.
Gå frakoblet med Player FM -appen!
Weak-To-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision
Fetch error
Hmmm there seems to be a problem fetching this series right now. Last successful fetch was on January 02, 2025 12:05 (
What now? This series will be checked again in the next hour. If you believe it should be working, please verify the publisher's feed link below is valid and includes actual episode links. You can contact support to request the feed be immediately fetched.
Manage episode 424744798 series 3498845
Widely used alignment techniques, such as reinforcement learning from human feedback (RLHF), rely on the ability of humans to supervise model behavior—for example, to evaluate whether a model faithfully followed instructions or generated safe outputs. However, future superhuman models will behave in complex ways too difficult for humans to reliably evaluate; humans will only be able to weakly supervise superhuman models. We study an analogy to this problem: can weak model supervision elicit the full capabilities of a much stronger model? We test this using a range of pretrained language models in the GPT-4 family on natural language processing (NLP), chess, and reward modeling tasks. We find that when we naively fine-tune strong pretrained models on labels generated by a weak model, they consistently perform better than their weak supervisors, a phenomenon we call weak-to-strong generalization. However, we are still far from recovering the full capabilities of strong models with naive fine-tuning alone, suggesting that techniques like RLHF may scale poorly to superhuman models without further work.
We find that simple methods can often significantly improve weak-to-strong generalization: for example, when fine-tuning GPT-4 with a GPT-2-level supervisor and an auxiliary confidence loss, we can recover close to GPT-3.5-level performance on NLP tasks. Our results suggest that it is feasible to make empirical progress today on a fundamental challenge of aligning superhuman models.
Source:
https://arxiv.org/pdf/2312.09390.pdf
Narrated for AI Safety Fundamentals by Perrin Walker
A podcast by BlueDot Impact.
Learn more on the AI Safety Fundamentals website.
Kapitler
1. Weak-To-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision (00:00:00)
2. ABSTRACT (00:00:18)
3. 1 INTRODUCTION (00:01:48)
4. 3 METHODOLOGY (00:09:29)
5. 4 MAIN RESULTS (00:14:44)
6. 4.1 TASKS (00:14:53)
7. 4.2 NAIVELY FINETUNING ON WEAK LABELS (00:17:01)
8. 4.3 IMPROVING WEAK-TO-STRONG GENERALIZATION IS TRACTABLE (00:20:10)
9. 4.3.1 BOOTSTRAPPING WITH INTERMEDIATE MODEL SIZES (00:20:30)
10. 4.3.2 AN AUXILIARY CONFIDENCE LOSS CAN DRAMATICALLY IMPROVE GENERALIZATION ON NLP TASKS (00:23:00)
11. 6 DISCUSSION (00:25:41)
12. 6.1 REMAINING DISANALOGIES (00:26:01)
13. 6.2 FUTURE WORK (00:29:00)
14. 6.2.1 CONCRETE PROBLEMS: ANALOGOUS SETUPS (00:29:26)
15. 6.2.2 CONCRETE PROBLEMS: SCALABLE METHODS (00:30:56)
16. 6.2.3 CONCRETE PROBLEMS: SCIENTIFIC UNDERSTANDING (00:32:33)
17. 6.3 CONCLUSION (00:33:58)
85 episoder
Fetch error
Hmmm there seems to be a problem fetching this series right now. Last successful fetch was on January 02, 2025 12:05 (
What now? This series will be checked again in the next hour. If you believe it should be working, please verify the publisher's feed link below is valid and includes actual episode links. You can contact support to request the feed be immediately fetched.
Manage episode 424744798 series 3498845
Widely used alignment techniques, such as reinforcement learning from human feedback (RLHF), rely on the ability of humans to supervise model behavior—for example, to evaluate whether a model faithfully followed instructions or generated safe outputs. However, future superhuman models will behave in complex ways too difficult for humans to reliably evaluate; humans will only be able to weakly supervise superhuman models. We study an analogy to this problem: can weak model supervision elicit the full capabilities of a much stronger model? We test this using a range of pretrained language models in the GPT-4 family on natural language processing (NLP), chess, and reward modeling tasks. We find that when we naively fine-tune strong pretrained models on labels generated by a weak model, they consistently perform better than their weak supervisors, a phenomenon we call weak-to-strong generalization. However, we are still far from recovering the full capabilities of strong models with naive fine-tuning alone, suggesting that techniques like RLHF may scale poorly to superhuman models without further work.
We find that simple methods can often significantly improve weak-to-strong generalization: for example, when fine-tuning GPT-4 with a GPT-2-level supervisor and an auxiliary confidence loss, we can recover close to GPT-3.5-level performance on NLP tasks. Our results suggest that it is feasible to make empirical progress today on a fundamental challenge of aligning superhuman models.
Source:
https://arxiv.org/pdf/2312.09390.pdf
Narrated for AI Safety Fundamentals by Perrin Walker
A podcast by BlueDot Impact.
Learn more on the AI Safety Fundamentals website.
Kapitler
1. Weak-To-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision (00:00:00)
2. ABSTRACT (00:00:18)
3. 1 INTRODUCTION (00:01:48)
4. 3 METHODOLOGY (00:09:29)
5. 4 MAIN RESULTS (00:14:44)
6. 4.1 TASKS (00:14:53)
7. 4.2 NAIVELY FINETUNING ON WEAK LABELS (00:17:01)
8. 4.3 IMPROVING WEAK-TO-STRONG GENERALIZATION IS TRACTABLE (00:20:10)
9. 4.3.1 BOOTSTRAPPING WITH INTERMEDIATE MODEL SIZES (00:20:30)
10. 4.3.2 AN AUXILIARY CONFIDENCE LOSS CAN DRAMATICALLY IMPROVE GENERALIZATION ON NLP TASKS (00:23:00)
11. 6 DISCUSSION (00:25:41)
12. 6.1 REMAINING DISANALOGIES (00:26:01)
13. 6.2 FUTURE WORK (00:29:00)
14. 6.2.1 CONCRETE PROBLEMS: ANALOGOUS SETUPS (00:29:26)
15. 6.2.2 CONCRETE PROBLEMS: SCALABLE METHODS (00:30:56)
16. 6.2.3 CONCRETE PROBLEMS: SCIENTIFIC UNDERSTANDING (00:32:33)
17. 6.3 CONCLUSION (00:33:58)
85 episoder
Alle episoder
×1 Introduction to Mechanistic Interpretability 11:45
1 Illustrating Reinforcement Learning from Human Feedback (RLHF) 22:32
1 Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback 32:19
1 Constitutional AI Harmlessness from AI Feedback 1:01:49
1 Empirical Findings Generalize Surprisingly Far 11:32
1 Two-Turn Debate Doesn’t Help Humans Answer Hard Reading Comprehension Questions 16:39
1 Least-To-Most Prompting Enables Complex Reasoning in Large Language Models 16:08
1 ABS: Scanning Neural Networks for Back-Doors by Artificial Brain Stimulation 16:08
1 Imitative Generalisation (AKA ‘Learning the Prior’) 18:14
Velkommen til Player FM!
Player FM scanner netter for høykvalitets podcaster som du kan nyte nå. Det er den beste podcastappen og fungerer på Android, iPhone og internett. Registrer deg for å synkronisere abonnement på flere enheter.