Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Video extraction is prone to race conditions #36

Open
cenodis opened this issue Dec 5, 2024 · 0 comments
Open

Video extraction is prone to race conditions #36

cenodis opened this issue Dec 5, 2024 · 0 comments

Comments

@cenodis
Copy link

cenodis commented Dec 5, 2024

tumblr-backup 1.0.5 via pip
tumblr-backup --save-video --save-audio keirastarlightdraconequus

After applying the fix in #35 tumblr-backup will properly execute yt-dlp to download videos.
However neither yt-dlp nor youtube-dl handle races when used in a multithreaded/multiprocessing environment [1]. This results in errors and (maybe?) file corruption.

This is especially frequent in blogs that reblog their own video posts close to each other, such as the customary 12-hour timezone reblog.

The best thing would be for tumblr-backup to handle these conflicts directly. It already resolves the target filename in get_youtube_url and could manage collisions using it as a key. This could also be used to filter out previously failed video URLs and help reduce unecessary yt-dlp calls.

[1] yt-dlp/yt-dlp#1918

Example errors:

ERROR: [youtube] f6ehAm9Ib-o: Video unavailableo 5199 (of 7288 expected)        
Unable to download video in post #146853269595                                  
ERROR: file:<REDACTED>/keirastarlightdraconequus/media/eFX04e9FuYM_@PitchandMagpieProductions_My_Secret_Weapon_MLP_Comic_Reading.f397.mp4: No such file or directory
ERROR: Unable to rename file: [Errno 2] No such file or directory: '<REDACTED>/keirastarlightdraconequus/media/k6xD1S4NfLU_@thewildcardperson_Molestia_vs_Discord_animation.f137.mp4.part' -> '<REDACTED>/keirastarlightdraconequus/media/k6xD1S4NfLU_@thewildcardperson_Molestia_vs_Discord_animation.f137.mp4'
ERROR: Unable to rename file: [Errno 2] No such file or directory: '<REDACTED>/keirastarlightdraconequus/media/k6xD1S4NfLU_@thewildcardperson_Molestia_vs_Discord_animation.f137.mp4.part' -> '<REDACTED>/keirastarlightdraconequus/media/k6xD1S4NfLU_@thewildcardperson_Molestia_vs_Discord_animation.f137.mp4'
ERROR: Unable to rename file: [Errno 2] No such file or directory: '<REDACTED>/keirastarlightdraconequus/media/k6xD1S4NfLU_@thewildcardperson_Molestia_vs_Discord_animation.f137.mp4.part' -> '<REDACTED>/keirastarlightdraconequus/media/k6xD1S4NfLU_@thewildcardperson_Molestia_vs_Discord_animation.f137.mp4'
ERROR: Unable to rename file: [Errno 2] No such file or directory: '<REDACTED>/keirastarlightdraconequus/media/k6xD1S4NfLU_@thewildcardperson_Molestia_vs_Discord_animation.f140.m4a.part' -> '<REDACTED>/keirastarlightdraconequus/media/k6xD1S4NfLU_@thewildcardperson_Molestia_vs_Discord_animation.f140.m4a'
ERROR: Unable to rename file: [Errno 2] No such file or directory: '<REDACTED>/keirastarlightdraconequus/media/k6xD1S4NfLU_@thewildcardperson_Molestia_vs_Discord_animation.f140.m4a.part' -> '<REDACTED>/keirastarlightdraconequus/media/k6xD1S4NfLU_@thewildcardperson_Molestia_vs_Discord_animation.f140.m4a'
ERROR: Unable to rename file: [Errno 2] No such file or directory: '<REDACTED>/keirastarlightdraconequus/media/k6xD1S4NfLU_@thewildcardperson_Molestia_vs_Discord_animation.f137.mp4.part' -> '<REDACTED>/keirastarlightdraconequus/media/k6xD1S4NfLU_@thewildcardperson_Molestia_vs_Discord_animation.f137.mp4'
ERROR: Unable to rename file: [Errno 2] No such file or directory: '<REDACTED>/keirastarlightdraconequus/media/k6xD1S4NfLU_@thewildcardperson_Molestia_vs_Discord_animation.f140.m4a.part' -> '<REDACTED>/keirastarlightdraconequus/media/k6xD1S4NfLU_@thewildcardperson_Molestia_vs_Discord_animation.f140.m4a'
ERROR: [Errno 2] No such file or directory: '<REDACTED>/keirastarlightdraconequus/media/k6xD1S4NfLU_@thewildcardperson_Molestia_vs_Discord_animation.temp.mp4' -> '<REDACTED>/keirastarlightdraconequus/media/k6xD1S4NfLU_@thewildcardperson_Molestia_vs_Discord_animation.mp4'
WARNING: Cannot update utime of file
WARNING: Cannot update utime of file
ERROR: [Errno 2] No such file or directory: '<REDACTED>/keirastarlightdraconequus/media/k6xD1S4NfLU_@thewildcardperson_Molestia_vs_Discord_animation.temp.mp4' -> '<REDACTED>/keirastarlightdraconequus/media/k6xD1S4NfLU_@thewildcardperson_Molestia_vs_Discord_animation.mp4'
ERROR: [Errno 2] No such file or directory: '<REDACTED>/keirastarlightdraconequus/media/k6xD1S4NfLU_@thewildcardperson_Molestia_vs_Discord_animation.temp.mp4' -> '<REDACTED>/keirastarlightdraconequus/media/k6xD1S4NfLU_@thewildcardperson_Molestia_vs_Discord_animation.mp4'
WARNING: Cannot update utime of file
ERROR: [Errno 2] No such file or directory: '<REDACTED>/keirastarlightdraconequus/media/k6xD1S4NfLU_@thewildcardperson_Molestia_vs_Discord_animation.temp.mp4' -> '<REDACTED>/keirastarlightdraconequus/media/k6xD1S4NfLU_@thewildcardperson_Molestia_vs_Discord_animation.mp4'
ERROR: [youtube] 5-bQE3WMzGA: Private video. Sign in if you've been granted access to this video. Use --cookies-from-browser or --cookies for the authentication. See  https://github.com/yt-dlp/yt-dlp/wiki/FAQ#how-do-i-pass-cookies-to-yt-dlp  for how to manually pass cookies. Also see  https://github.com/yt-dlp/yt-dlp/wiki/Extractors#exporting-youtube-cookies  for tips on effectively exporting YouTube cookies
Unable to download video in post #144696447150
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant