Artwork

Innhold levert av Patrick Wheeler and Jason Gauci, Patrick Wheeler, and Jason Gauci. Alt podcastinnhold, inkludert episoder, grafikk og podcastbeskrivelser, lastes opp og leveres direkte av Patrick Wheeler and Jason Gauci, Patrick Wheeler, and Jason Gauci eller deres podcastplattformpartner. Hvis du tror at noen bruker det opphavsrettsbeskyttede verket ditt uten din tillatelse, kan du følge prosessen skissert her https://no.player.fm/legal.
Player FM - Podcast-app
Gå frakoblet med Player FM -appen!

185: Workflow Orchestrators

1:32:02
 
Del
 

Manage episode 517701083 series 8393
Innhold levert av Patrick Wheeler and Jason Gauci, Patrick Wheeler, and Jason Gauci. Alt podcastinnhold, inkludert episoder, grafikk og podcastbeskrivelser, lastes opp og leveres direkte av Patrick Wheeler and Jason Gauci, Patrick Wheeler, and Jason Gauci eller deres podcastplattformpartner. Hvis du tror at noen bruker det opphavsrettsbeskyttede verket ditt uten din tillatelse, kan du følge prosessen skissert her https://no.player.fm/legal.

Intro topic: Asymmetric Returns

News/Links:

Book of the Show

Patreon Plug https://www.patreon.com/programmingthrowdown?ty=h

Tool of the Show

Topic: Workflow Orchestrators

  • Why
    • Batch jobs (embarrassingly parallel)
    • Long-running tasks (e.g. transcoding video)
    • Checkpointing/resuming
  • How
    • Message Queues
    • Containerization
    • Worker Pools & Autoscaling
    • History & Backfill
  • Steps to run workflows:
    • Containerize the workflow definition and send to the cloud
    • Containerize all the individual tasks
    • Submit job(s)
  • Examples
    • Airflow
      • Legacy but dominant
    • Dagster
      • Great UX for python developers
    • Temporal: https://temporal.io/
      • The new hotness
    • Ray
      • Low-level but very powerful
    • Kubeflow
      • Designed for ML workflows, integrated dashboard

★ Support this podcast on Patreon ★
  continue reading

186 episoder

Artwork

185: Workflow Orchestrators

Programming Throwdown

7,601 subscribers

published

iconDel
 
Manage episode 517701083 series 8393
Innhold levert av Patrick Wheeler and Jason Gauci, Patrick Wheeler, and Jason Gauci. Alt podcastinnhold, inkludert episoder, grafikk og podcastbeskrivelser, lastes opp og leveres direkte av Patrick Wheeler and Jason Gauci, Patrick Wheeler, and Jason Gauci eller deres podcastplattformpartner. Hvis du tror at noen bruker det opphavsrettsbeskyttede verket ditt uten din tillatelse, kan du følge prosessen skissert her https://no.player.fm/legal.

Intro topic: Asymmetric Returns

News/Links:

Book of the Show

Patreon Plug https://www.patreon.com/programmingthrowdown?ty=h

Tool of the Show

Topic: Workflow Orchestrators

  • Why
    • Batch jobs (embarrassingly parallel)
    • Long-running tasks (e.g. transcoding video)
    • Checkpointing/resuming
  • How
    • Message Queues
    • Containerization
    • Worker Pools & Autoscaling
    • History & Backfill
  • Steps to run workflows:
    • Containerize the workflow definition and send to the cloud
    • Containerize all the individual tasks
    • Submit job(s)
  • Examples
    • Airflow
      • Legacy but dominant
    • Dagster
      • Great UX for python developers
    • Temporal: https://temporal.io/
      • The new hotness
    • Ray
      • Low-level but very powerful
    • Kubeflow
      • Designed for ML workflows, integrated dashboard

★ Support this podcast on Patreon ★
  continue reading

186 episoder

Alle episoder

×
 
Loading …

Velkommen til Player FM!

Player FM scanner netter for høykvalitets podcaster som du kan nyte nå. Det er den beste podcastappen og fungerer på Android, iPhone og internett. Registrer deg for å synkronisere abonnement på flere enheter.

 

Hurtigreferanseguide

Copyright 2025 | Personvern | Vilkår for bruk | | opphavsrett
Lytt til dette showet mens du utforsker
Spill