Hot And Cold Data With Apache Kafka, Tiered Storage, And Iceberg Data (R)evolution podcast

Hot and cold data with Apache Kafka, Tiered Storage, and Iceberg

3M ago 48:58

Del

Innhold levert av Aiven. Alt podcastinnhold, inkludert episoder, grafikk og podcastbeskrivelser, lastes opp og leveres direkte av Aiven eller deres podcastplattformpartner. Hvis du tror at noen bruker det opphavsrettsbeskyttede verket ditt uten din tillatelse, kan du følge prosessen skissert her https://no.player.fm/legal.

Utilizing the true potential of data streaming is key to business success.

In this Data (R)evolution episode, we're joined by Josep Prat and Filip Yonov to dive into the transformative features of Apache Kafka and its evolving role in data architecture. They discuss the critical importance of collaboration and feedback in enhancing Kafka's capabilities, the future of "lake house" technology, exciting updates from the Open Source Program Office (OSPO), and the importance of Kafka's readiness to support evolving data formats—making it a backbone for modern data ecosystems.

Key Takeaways:

Community collaboration and contribution are essential for the continuous improvement and testing of Apache Kafka's capabilities
The evolution of Apache Kafka into a more versatile platform, combined with object storage and open table formats, can significantly enhance real-time data streaming, analytics, and the future of "lake house" technology
Tiered storage in Kafka facilitates more efficient and cost-effective data management by decoupling storage from computing

Resources:

Watch the full interview on our YouTube: https://www.youtube.com/@Aiven_io
Check out our website for more information: https://aiven.io/
Check out Aiven AI Database Optimizer
Want to be on our mailing list? Sign up here: https://aiven.io/resources
Follow us on LinkedIn: https://www.linkedin.com/company/aiven/
Sign up for our newsletter for more insights on this topic: https://aiven.io/newsletter
Connect with Filip Yonov on LinkedIn: https://www.linkedin.com/in/filipyonov/
Connect with Josep Prat on LinkedIn: https://www.linkedin.com/in/jlprat/

Timestamps:

[05:49] Kafka servers have theoretical storage limits

[09:29] Test storage proposal process for Apache Kafka

[17:38] LinkedIn conducted an experiment merging Xcode versions

[22:11] Data lake evolving into lake house architectures

[25:00] Broker pushes data to remote storage, plugin handles retrieval and format translation

[26:40] Kafka excels at high-speed, high-volume data

[32:18] Kafka data consumption evolving with new options

[40:19] Managing metadata for conversion on community level

[47:45] Kafka's potential as a widely used API

11 episoder