Looks like the publisher may have taken this series offline or changed its URL. Please contact support if you believe it should be working, the feed URL is invalid, or you have any other concerns about it.
Gå frakoblet med Player FM -appen!
Discovering Latent Knowledge in Language Models Without Supervision
Fetch error
Hmmm there seems to be a problem fetching this series right now. Last successful fetch was on January 02, 2025 12:05 (
What now? This series will be checked again in the next hour. If you believe it should be working, please verify the publisher's feed link below is valid and includes actual episode links. You can contact support to request the feed be immediately fetched.
Manage episode 424087972 series 3498845
Abstract:
Existing techniques for training language models can be misaligned with the truth: if we train models with imitation learning, they may reproduce errors that humans make; if we train them to generate text that humans rate highly, they may output errors that human evaluators can't detect. We propose circumventing this issue by directly finding latent knowledge inside the internal activations of a language model in a purely unsupervised way. Specifically, we introduce a method for accurately answering yes-no questions given only unlabeled model activations. It works by finding a direction in activation space that satisfies logical consistency properties, such as that a statement and its negation have opposite truth values. We show that despite using no supervision and no model outputs, our method can recover diverse knowledge represented in large language models: across 6 models and 10 question-answering datasets, it outperforms zero-shot accuracy by 4\\% on average. We also find that it cuts prompt sensitivity in half and continues to maintain high accuracy even when models are prompted to generate incorrect answers. Our results provide an initial step toward discovering what language models know, distinct from what they say, even when we don't have access to explicit ground truth labels.
Original text:
https://arxiv.org/abs/2212.03827
Narrated for AI Safety Fundamentals by Perrin Walker of TYPE III AUDIO.
---
A podcast by BlueDot Impact.
Learn more on the AI Safety Fundamentals website.
Kapitler
1. Discovering Latent Knowledge in Language Models Without Supervision (00:00:00)
2. ABSTRACT (00:00:12)
3. 1 INTRODUCTION (00:01:29)
4. 2 PROBLEM STATEMENT AND FRAMEWORK (00:06:32)
5. 2.1 PROBLEM: DISCOVERING LATENT KNOWLEDGE (00:07:04)
6. 2.2 METHOD: CONTRAST-CONSISTENT SEARCH (00:08:31)
7. Constructing contrast pairs. (00:10:16)
8. Feature extraction and normalization. (00:11:43)
9. Inference. (00:15:58)
10. 3 RESULTS (00:17:04)
11. 3.1 EXPERIMENTAL SETUP (00:17:07)
12. 3.2 EVALUATING CCS (00:23:41)
13. 3.2.1 CCS OUTPERFORMS ZERO-SHOT (00:23:44)
14. 3.2.2 CCS IS ROBUST TO MISLEADING PROMPTS (00:25:17)
15. 3.3 ANALYZING CCS (00:26:41)
16. 3.3.1 CCS FINDS A TASK-AGNOSTIC REPRESENTATION OF TRUTH (00:27:12)
17. 3.3.2 CCS DOES NOT JUST RECOVER MODEL OUTPUTS (00:30:00)
18. 3.3.3 TRUTH IS A SALIENT FEATURE (00:31:45)
85 episoder
Fetch error
Hmmm there seems to be a problem fetching this series right now. Last successful fetch was on January 02, 2025 12:05 (
What now? This series will be checked again in the next hour. If you believe it should be working, please verify the publisher's feed link below is valid and includes actual episode links. You can contact support to request the feed be immediately fetched.
Manage episode 424087972 series 3498845
Abstract:
Existing techniques for training language models can be misaligned with the truth: if we train models with imitation learning, they may reproduce errors that humans make; if we train them to generate text that humans rate highly, they may output errors that human evaluators can't detect. We propose circumventing this issue by directly finding latent knowledge inside the internal activations of a language model in a purely unsupervised way. Specifically, we introduce a method for accurately answering yes-no questions given only unlabeled model activations. It works by finding a direction in activation space that satisfies logical consistency properties, such as that a statement and its negation have opposite truth values. We show that despite using no supervision and no model outputs, our method can recover diverse knowledge represented in large language models: across 6 models and 10 question-answering datasets, it outperforms zero-shot accuracy by 4\\% on average. We also find that it cuts prompt sensitivity in half and continues to maintain high accuracy even when models are prompted to generate incorrect answers. Our results provide an initial step toward discovering what language models know, distinct from what they say, even when we don't have access to explicit ground truth labels.
Original text:
https://arxiv.org/abs/2212.03827
Narrated for AI Safety Fundamentals by Perrin Walker of TYPE III AUDIO.
---
A podcast by BlueDot Impact.
Learn more on the AI Safety Fundamentals website.
Kapitler
1. Discovering Latent Knowledge in Language Models Without Supervision (00:00:00)
2. ABSTRACT (00:00:12)
3. 1 INTRODUCTION (00:01:29)
4. 2 PROBLEM STATEMENT AND FRAMEWORK (00:06:32)
5. 2.1 PROBLEM: DISCOVERING LATENT KNOWLEDGE (00:07:04)
6. 2.2 METHOD: CONTRAST-CONSISTENT SEARCH (00:08:31)
7. Constructing contrast pairs. (00:10:16)
8. Feature extraction and normalization. (00:11:43)
9. Inference. (00:15:58)
10. 3 RESULTS (00:17:04)
11. 3.1 EXPERIMENTAL SETUP (00:17:07)
12. 3.2 EVALUATING CCS (00:23:41)
13. 3.2.1 CCS OUTPERFORMS ZERO-SHOT (00:23:44)
14. 3.2.2 CCS IS ROBUST TO MISLEADING PROMPTS (00:25:17)
15. 3.3 ANALYZING CCS (00:26:41)
16. 3.3.1 CCS FINDS A TASK-AGNOSTIC REPRESENTATION OF TRUTH (00:27:12)
17. 3.3.2 CCS DOES NOT JUST RECOVER MODEL OUTPUTS (00:30:00)
18. 3.3.3 TRUTH IS A SALIENT FEATURE (00:31:45)
85 episoder
Alle episoder
×1 Introduction to Mechanistic Interpretability 11:45
1 Illustrating Reinforcement Learning from Human Feedback (RLHF) 22:32
1 Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback 32:19
1 Constitutional AI Harmlessness from AI Feedback 1:01:49
1 Empirical Findings Generalize Surprisingly Far 11:32
1 Two-Turn Debate Doesn’t Help Humans Answer Hard Reading Comprehension Questions 16:39
1 Least-To-Most Prompting Enables Complex Reasoning in Large Language Models 16:08
1 ABS: Scanning Neural Networks for Back-Doors by Artificial Brain Stimulation 16:08
1 Imitative Generalisation (AKA ‘Learning the Prior’) 18:14
Velkommen til Player FM!
Player FM scanner netter for høykvalitets podcaster som du kan nyte nå. Det er den beste podcastappen og fungerer på Android, iPhone og internett. Registrer deg for å synkronisere abonnement på flere enheter.