Welcome back to the Airflow Podcast.This week, we met up with Ben Wisegarver, a staff data scientist at Reddit who runs their data warehousing and data engineering functions.Reddit users generate petabytes of data every day that needs to be processed, stored, and analyzed by a wide breadth of backend services. Our conversation with Ben touches on e…

Welcome back to the Airflow Podcast.This week, we met up with Albert Franzi and Carlos Escura from Typeform. Typeform is a tool that allows you to build beautiful interactive forms that you can use for a wide variety of use cases, including customer surveys, employee engagement, product feedback, and market research to name a few. In our conversati…

After a bit of a break, we're back with the third official episode bundle of The Airflow Podcast. In this batch, we'll get a little bit deeper with current Airflow users and maintainers on core fundamental concepts in data engineering, architectures for operating modern data platforms at scale, and the process of maintaining and operating Airflow, …

This week, we linked up with Airflow release manager, core committer, and Astronomer platform engineer Ash Berlin-Taylor to discuss the Airflow 2.0 roadmap [1]. There is some great stuff in the works around performance, autoscaling, and usability that we're excited about. In this episode, Ash lends his thoughts on the design, implementation, and va…

This week, we had the pleasure of meeting up with Jarek Potiuk, Principal Software Engineer at Polidea and Apache Airflow committer, to discuss his most recent contribution to the community, Airflow Breeze. Jarek deeply values developer productivity and realized while building a team of Airflow committers that, in order to open a PR on the project,…

This episode kicks off season 2 of The Airflow Podcast. In this next season, we'll focus on the future of Airflow and chat with leading members of the community to paint a picture of what's to come. We're pumped to be diving back into this project and look forward to the great conversations we have lined up.This week, we chatted with James Malone, …

This week, we met up with Ash Berlin-Taylor to discuss the recent 1.10 release, what it's like to be a release manager for an open source project, Airflow's bid to graduate from incubating status, and the next phase of Airflow project development.As mentioned in our podcast intro, we at Astronomer are hiring Data Engineers who are passionate about …

This time, we met up with WePay's Joy Gao to talk through her work on the RBAC components in the recent Airflow 1.10 release. We dove deep into what inspired her work and took some time to discuss what it's like to be a woman contributing to a predominately male open-source community. Hope you enjoy!If you'd like to get started using Airflow in you…

In this episode, we dove into the relationship between Airflow and Kuberenetes and interviewed Daniel Imberman, Senior Software Engineer at Bloomberg (1:30), and Greg Neiheisel, CTO here at Astronomer (37:31). Daniel has done most of the work on the Kubernetes executor for Airflow and Greg plans to take on a chunk of the development going forward, …

This week, we’ll examine conversations with both old guests and new to paint a comprehensive picture of Airflow’s pain points. While we still undoubtedly believe that Airflow is the future of ETL, it’s important to acknowledge that any incubating project will have issues, and bringing those issues to the forefront of the community’s attention will …

On this episode, we linked up with Erik Bernhardsson (@erikbern), creator of Luigi and CTO of Better Mortgage. We chatted about everything from the motivations behind Luigi's creation and his current thoughts on Airflow- we hope you enjoy!Check out:- Erik's blog at erikbern.com- Our open-source library of Airflow plugins at github.com/airflow-plugi…

In this episode, we dive into Airflow Best Practices and include longer portions of interviews with Alan Cruickshank (1:30), Business Insights and Data Manager at Tails.com, Chris Riccomini (7:27), Principal Software Engineer at WePay, and Bolke de Bruin(31:45), Head of Advanced Analytics Technology at ING. Hope you enjoy!We're still working to get…

Episode 2 of The Airflow Podcast is here to discuss six specific use cases that we’ve seen for Apache Airflow. Here’s the lineup:Patrick Atwater (@patwater), Water Data Projects Manager at ARGO Labs: 2:03-5:35Maksime Pecherskiy (@mrmaksimize), CDO of San Diego: 5:35-23:06Scott Halgrim (@shalgrim), Data Engineer at Zapier: 23:06-27:27Bolke de Bruin …

For the first episode of the Airflow Podcast, we met up with Maxime Beauchemin, creator of Airflow, to explore the motivations behind its creation and the problems it was designed to solve. We asked Maxime for his definition of Airflow, the design principles behind hook/operator use, and his vision for the project.Speaker list:Pete DeJoy - Product …

A sneak peek at our upcoming podcast about Apache Airflow.Featured in this clip (in order of appearance):Pete DeJoy - Product Specialist at AstronomerPatrick Atwater - Water Data Projects Manager at ARGO LabsMaksime Pecherskiy - Chief Data Officer of the City of San DiegoBolke de Bruin - Head of Advanced Analytics at ING…