DataLoader With Multiple Workers Leaks Memory PyTorch Developer podcast

Artwork

Tech PyTorch Edward Yang Team PyTorch Deep Learning Machine Learning

Innhold levert av PyTorch, Edward Yang, and Team PyTorch. Alt podcastinnhold, inkludert episoder, grafikk og podcastbeskrivelser, lastes opp og leveres direkte av PyTorch, Edward Yang, and Team PyTorch eller deres podcastplattformpartner. Hvis du tror at noen bruker det opphavsrettsbeskyttede verket ditt uten din tillatelse, kan du følge prosessen skissert her https://no.player.fm/legal.

PyTorch Developer Podcast « »
DataLoader with multiple workers leaks memory

4+ y ago 16:38

Del

MP3•Episoder hjem

Innhold levert av PyTorch, Edward Yang, and Team PyTorch. Alt podcastinnhold, inkludert episoder, grafikk og podcastbeskrivelser, lastes opp og leveres direkte av PyTorch, Edward Yang, and Team PyTorch eller deres podcastplattformpartner. Hvis du tror at noen bruker det opphavsrettsbeskyttede verket ditt uten din tillatelse, kan du følge prosessen skissert her https://no.player.fm/legal.

Today I'm going to talk about a famous issue in PyTorch, DataLoader with num_workers > 0 causes memory leak (https://github.com/pytorch/pytorch/issues/13246). This bug is a good opportunity to talk about DataSet/DataLoader design in PyTorch, fork and copy-on-write memory in Linux and Python reference counting; you have to know about all of these things to understand why this bug occurs, but once you do, it also explains why the workarounds help.

Further reading.

A nice summary of the full issue https://github.com/pytorch/pytorch/issues/13246#issuecomment-905703662
DataLoader architecture RFC https://github.com/pytorch/pytorch/issues/49440
Cinder Python https://github.com/facebookincubator/cinder

… continue reading

83 episoder

#Tech #PyTorch #Edward Yang #Team PyTorch #Deep Learning #Machine Learning

Artwork

DataLoader with multiple workers leaks memory

PyTorch Developer Podcast

26 subscribers

published 4+ y ago

Del

MP3•Episoder hjem

Innhold levert av PyTorch, Edward Yang, and Team PyTorch. Alt podcastinnhold, inkludert episoder, grafikk og podcastbeskrivelser, lastes opp og leveres direkte av PyTorch, Edward Yang, and Team PyTorch eller deres podcastplattformpartner. Hvis du tror at noen bruker det opphavsrettsbeskyttede verket ditt uten din tillatelse, kan du følge prosessen skissert her https://no.player.fm/legal.

Today I'm going to talk about a famous issue in PyTorch, DataLoader with num_workers > 0 causes memory leak (https://github.com/pytorch/pytorch/issues/13246). This bug is a good opportunity to talk about DataSet/DataLoader design in PyTorch, fork and copy-on-write memory in Linux and Python reference counting; you have to know about all of these things to understand why this bug occurs, but once you do, it also explains why the workarounds help.

Further reading.

A nice summary of the full issue https://github.com/pytorch/pytorch/issues/13246#issuecomment-905703662
DataLoader architecture RFC https://github.com/pytorch/pytorch/issues/49440
Cinder Python https://github.com/facebookincubator/cinder

… continue reading

83 episoder

#Tech #PyTorch #Edward Yang #Team PyTorch #Deep Learning #Machine Learning

Όλα τα επεισόδια

×

Velkommen til Player FM!

Player FM scanner netter for høykvalitets podcaster som du kan nyte nå. Det er den beste podcastappen og fungerer på Android, iPhone og internett. Registrer deg for å synkronisere abonnement på flere enheter.

Lytt til 500+ tema

Lytt til dette showet mens du utforsker