Limits of Embeddings: Out-of-Domain Data, Long Context, Finetuning (and How We're Fixing It) | S2 E5

How AI Is Built

Text embeddings have limitations when it comes to handling long documents and out-of-domain data.

Today, we are talking to Nils Reimers. He is one of the researchers who kickstarted the field of dense embeddings, developed sentence transformers, started HuggingFace’s Neural Search team and now leads the development of search foundational models at Cohere. Tbh, he has too many accolades to count off here.

We talk about the main limitations of embeddings:

• Failing out of domain
• Struggling with long documents
• Very hard to debug
• Hard to find formalize what actually is similar

Are you still not sure whether to listen? Here are some teasers:

• Interpreting embeddings can be challenging, and current models are not easily explainable.
• Fine-tuning is necessary to adapt embeddings to specific domains, but it requires careful consideration of the data and objectives.
• Re-ranking is an effective approach to handle long documents and incorporate additional factors like recency and trustworthiness.
• The future of embeddings lies in addressing scalability issues and exploring new research directions.

Nils Reimers:

• LinkedIn
• X (Twitter)
• Website
• Cohere

Nicolay Gerold:

• ⁠LinkedIn⁠
• ⁠X (Twitter)

text embeddings, limitations, long documents, interpretation, fine-tuning, re-ranking, future research

00:00 Introduction and Guest Introduction 00:43 Early Work with BERT and Argument Mining 02:24 Evolution and Innovations in Embeddings 03:39 Constructive Learning and Hard Negatives 05:17 Training and Fine-Tuning Embedding Models 12:48 Challenges and Limitations of Embeddings 18:16 Adapting Embeddings to New Domains 22:41 Handling Long Documents and Re-Ranking 31:08 Combining Embeddings with Traditional ML 45:16 Conclusion and Upcoming Episodes