PeerTube icon indicating copy to clipboard operation
PeerTube copied to clipboard
verified publisher icon Chocobozzz

New YouTube videos don't show up when using the new synchronization functionality

Open emansom opened this issue 3 years ago • 2 comments

Describe the current behavior

New YouTube videos don't show up when triggered by the channel synchronization scheduler.

Only when manually clicking the synchronize channel button via the UI it fetches the new YouTube videos.

Steps to reproduce

  1. Setup PeerTube using latest develop branch (2022-09-13)
  2. Synchronize a full YouTube channel and wait until all jobs completed
  3. Publish a new video on the YouTube channel
  4. Observe PeerTube never picking it up automatically

Describe the expected behavior

PeerTube picking up new videos from YouTube automatically every hour via the channel synchronization scheduler.

Additional information

  • PeerTube instance:

    • URL: https://pcbu.nl
    • Version: latest code in develop branch (2022-09-12)
    • NodeJS version: v16.17.0
    • Ffmpeg version: 4.3.4-0+deb11u1

  • Browser name, version and platforms on which you could reproduce the bug: irrelevant

  • Link to browser console log if relevant: irrelevant

  • Link to server log if relevant (journalctl or /var/www/peertube/storage/logs/): https://gist.github.com/emansom/8fcdcf5ae31e47c245c798e25b7b2ee3

emansom avatar Sep 12 '22 13:09 emansom

Hi,

Please enable debug logs and check debug: Running VideoChannelSyncLatestScheduler.internalExecute is displayed. Then please paste channels sync related logs

Chocobozzz avatar Sep 14 '22 13:09 Chocobozzz

Hi,

Please enable debug logs and check debug: Running VideoChannelSyncLatestScheduler.internalExecute is displayed. Then please paste channels sync related logs

Full log: https://gist.github.com/emansom/8fcdcf5ae31e47c245c798e25b7b2ee3

The VideoChannelSyncLatestScheduler fails to synchronize https://www.youtube.com/watch?v=Mnu6dXmhjCg and https://www.youtube.com/watch?v=vQCZUK9zMYQ for PeerTube channel dpc_shorts (PoppenCast Clips on YouTube).

It runs the yt-dlp commands to list data, but never actually downloads nor publishes the videos.

[pcbu.nl:443] 9/14/2022, 2:26:59 PM.530 debug: Run youtube-dl command. {
  "command": "/usr/bin/python3 /data/bin/yt-dlp --dump-json -f bestvideo[vcodec!*=av01][vcodec!*=vp9.2]+bestaudio/best[vcodec!*=av01][vcodec!*=vp9.2]/bestvideo[ext=mp4]+bestaudio[ext=m4a]/best https://www.youtube.com/watch?v=Mnu6dXmhjCg"
}
[pcbu.nl:443] 9/14/2022, 2:26:59 PM.536 debug: Executed SQL request - Executing (default): SELECT "VideoImportModel"."id", "VideoImportModel"."targetUrl", "VideoImportModel"."magnetUri", "VideoImportModel"."torrentName", "VideoImportModel"."state", "VideoImportModel"."error", "VideoImportModel"."userId", "VideoImportModel"."videoId", "VideoImportModel"."videoChannelSyncId", "VideoImportModel"."createdAt", "VideoImportModel"."updatedAt", "Video"."id" AS "Video.id", "Video"."uuid" AS "Video.uuid", "Video"."name" AS "Video.name", "Video"."category" AS "Video.category", "Video"."licence" AS "Video.licence", "Video"."language" AS "Video.language", "Video"."privacy" AS "Video.privacy", "Video"."nsfw" AS "Video.nsfw", "Video"."description" AS "Video.description", "Video"."support" AS "Video.support", "Video"."duration" AS "Video.duration", "Video"."views" AS "Video.views", "Video"."likes" AS "Video.likes", "Video"."dislikes" AS "Video.dislikes", "Video"."remote" AS "Video.remote", "Video"."isLive" AS "Video.isLive", "Video"."url" AS "Video.url", "Video"."commentsEnabled" AS "Video.commentsEnabled", "Video"."downloadEnabled" AS "Video.downloadEnabled", "Video"."waitTranscoding" AS "Video.waitTranscoding", "Video"."state" AS "Video.state", "Video"."publishedAt" AS "Video.publishedAt", "Video"."originallyPublishedAt" AS "Video.originallyPublishedAt", "Video"."channelId" AS "Video.channelId", "Video"."createdAt" AS "Video.createdAt", "Video"."updatedAt" AS "Video.updatedAt" FROM "videoImport" AS "VideoImportModel" INNER JOIN "video" AS "Video" ON "VideoImportModel"."videoId" = "Video"."id" AND "Video"."channelId" = 3 WHERE "VideoImportModel"."targetUrl" = 'https://www.youtube.com/watch?v=vQCZUK9zMYQ' AND "VideoImportModel"."state" IN (1, 6, 2) LIMIT 1;
[pcbu.nl:443] 9/14/2022, 2:27:01 PM.233 debug: Run youtube-dl command. {
  "command": "/usr/bin/python3 /data/bin/yt-dlp --dump-json -f bestvideo[vcodec!*=av01][vcodec!*=vp9.2]+bestaudio/best[vcodec!*=av01][vcodec!*=vp9.2]/bestvideo[ext=mp4]+bestaudio[ext=m4a]/best https://www.youtube.com/watch?v=vQCZUK9zMYQ"
}
[pcbu.nl:443] 9/14/2022, 2:27:01 PM.243 info: Processing after video channel import in job b0f84468-a940-4b79-86ef-e13e013947c9.
[pcbu.nl:443] 9/14/2022, 2:27:01 PM.244 debug: Executed SQL request - Executing (default): SELECT "VideoChannelSyncModel"."id", "VideoChannelSyncModel"."externalChannelUrl", "VideoChannelSyncModel"."videoChannelId", "VideoChannelSyncModel"."state", "VideoChannelSyncModel"."lastSyncAt", "VideoChannelSyncModel"."createdAt", "VideoChannelSyncModel"."updatedAt", "VideoChannel"."id" AS "VideoChannel.id", "VideoChannel"."name" AS "VideoChannel.name", "VideoChannel"."description" AS "VideoChannel.description", "VideoChannel"."support" AS "VideoChannel.support", "VideoChannel"."actorId" AS "VideoChannel.actorId", "VideoChannel"."accountId" AS "VideoChannel.accountId", "VideoChannel"."createdAt" AS "VideoChannel.createdAt", "VideoChannel"."updatedAt" AS "VideoChannel.updatedAt", "VideoChannel->Actor"."id" AS "VideoChannel.Actor.id", "VideoChannel->Actor"."type" AS "VideoChannel.Actor.type", "VideoChannel->Actor"."preferredUsername" AS "VideoChannel.Actor.preferredUsername", "VideoChannel->Actor"."url" AS "VideoChannel.Actor.url", "VideoChannel->Actor"."publicKey" AS "VideoChannel.Actor.publicKey", "VideoChannel->Actor"."privateKey" AS "VideoChannel.Actor.privateKey", "VideoChannel->Actor"."followersCount" AS "VideoChannel.Actor.followersCount", "VideoChannel->Actor"."followingCount" AS "VideoChannel.Actor.followingCount", "VideoChannel->Actor"."inboxUrl" AS "VideoChannel.Actor.inboxUrl", "VideoChannel->Actor"."outboxUrl" AS "VideoChannel.Actor.outboxUrl", "VideoChannel->Actor"."sharedInboxUrl" AS "VideoChannel.Actor.sharedInboxUrl", "VideoChannel->Actor"."followersUrl" AS "VideoChannel.Actor.followersUrl", "VideoChannel->Actor"."followingUrl" AS "VideoChannel.Actor.followingUrl", "VideoChannel->Actor"."remoteCreatedAt" AS "VideoChannel.Actor.remoteCreatedAt", "VideoChannel->Actor"."serverId" AS "VideoChannel.Actor.serverId", "VideoChannel->Actor"."createdAt" AS "VideoChannel.Actor.createdAt", "VideoChannel->Actor"."updatedAt" AS "VideoChannel.Actor.updatedAt", "VideoChannel->Actor->Server"."id" AS "VideoChannel.Actor.Server.id", "VideoChannel->Actor->Server"."host" AS "VideoChannel.Actor.Server.host", "VideoChannel->Actor->Server"."redundancyAllowed" AS "VideoChannel.Actor.Server.redundancyAllowed", "VideoChannel->Actor->Server"."createdAt" AS "VideoChannel.Actor.Server.createdAt", "VideoChannel->Actor->Server"."updatedAt" AS "VideoChannel.Actor.Server.updatedAt", "VideoChannel->Actor->Avatars"."id" AS "VideoChannel.Actor.Avatars.id", "VideoChannel->Actor->Avatars"."filename" AS "VideoChannel.Actor.Avatars.filename", "VideoChannel->Actor->Avatars"."height" AS "VideoChannel.Actor.Avatars.height", "VideoChannel->Actor->Avatars"."width" AS "VideoChannel.Actor.Avatars.width", "VideoChannel->Actor->Avatars"."fileUrl" AS "VideoChannel.Actor.Avatars.fileUrl", "VideoChannel->Actor->Avatars"."onDisk" AS "VideoChannel.Actor.Avatars.onDisk", "VideoChannel->Actor->Avatars"."type" AS "VideoChannel.Actor.Avatars.type", "VideoChannel->Actor->Avatars"."actorId" AS "VideoChannel.Actor.Avatars.actorId", "VideoChannel->Actor->Avatars"."createdAt" AS "VideoChannel.Actor.Avatars.createdAt", "VideoChannel->Actor->Avatars"."updatedAt" AS "VideoChannel.Actor.Avatars.updatedAt" FROM "videoChannelSync" AS "VideoChannelSyncModel" INNER JOIN "videoChannel" AS "VideoChannel" ON "VideoChannelSyncModel"."videoChannelId" = "VideoChannel"."id" INNER JOIN "actor" AS "VideoChannel->Actor" ON "VideoChannel"."actorId" = "VideoChannel->Actor"."id" LEFT OUTER JOIN "server" AS "VideoChannel->Actor->Server" ON "VideoChannel->Actor"."serverId" = "VideoChannel->Actor->Server"."id" LEFT OUTER JOIN "actorImage" AS "VideoChannel->Actor->Avatars" ON "VideoChannel->Actor"."id" = "VideoChannel->Actor->Avatars"."actorId" AND "VideoChannel->Actor->Avatars"."type" = 1 WHERE "VideoChannelSyncModel"."id" = 2;
[pcbu.nl:443] 9/14/2022, 2:27:01 PM.247 info: Finished synchronizing "dpc_shorts" successfully. {
  "successes": 0
}

emansom avatar Sep 14 '22 14:09 emansom

Thanks, it should be fixed by https://github.com/Chocobozzz/PeerTube/commit/3204f4d17dec79413a94eae8a71b72e0c7ec7ba5

Chocobozzz avatar Sep 16 '22 07:09 Chocobozzz

@Chocobozzz That fixed it! Thanks! 👍🏻

emansom avatar Sep 16 '22 10:09 emansom

I made a mistake in the previous commit, that should be fixed by https://github.com/Chocobozzz/PeerTube/commit/9a3a23a834679e072c15960903e05e667ef14481

Chocobozzz avatar Sep 16 '22 11:09 Chocobozzz