YouTube-Agent.bundle
YouTube-Agent.bundle copied to clipboard
ZeroQI
Channel metadata and thumbnail not appearing but episode metadata and thumbnails work fine
Hello there, excellent work with the agent. Unfortunately it seems there's an issue with the channel metadata and thumbnail not appearing but episode metadata and thumbnails work fine.
- My output template is
--output "/Media/YouTube/%(uploader)s [%(channel_id)s]/%(upload_date>%Y-%m-%d)s - %(title)s [%(id)s].%(ext)s"which resolves to/Media/YouTube/belle delphine [UCXvKUavCtDOlA8bT1i2tI3w]/2020-06-17 - I'M BACK -belle delphine [TL470fJMi7w].mkv - I'm using my own API key
- Using latest YouTube Agent and ASS Scanner on latest PMS on Windows 11 (latest as today)
- Logs attached (tried multiple wipes, reinstalls and restarts)
Screenshot of "Channel metadata and thumbnail not appearing"
Some channel metadata gets scraped apparently
Screenshot of "episode metadata and thumbnails working fine"
Log file with API key redacted
com.plexapp.agents.youtube.log
No search log in logs provided No Plex token ID file Language xn for library = Chinese?
INFO (logkit:16) - Place correct Plex token in X-Plex-Token.id file in logs folder or in PLEX_LIBRARY_URL variable to have a log per library - https://support.plex.tv/hc/en-us/articles/204059436-Finding-your-account-token-X-Plex-TokenHTTP Error 401: Unauthorized [ ] dir: "D:\Media\YouTube\belle delphine [UCXvKUavCtDOlA8bT1i2tI3w]" INFO (logkit:16) - No GUID so random folder
(https://github.com/ZeroQI/YouTube-Agent.bundle/blob/master/Contents/Code/init.py#L533)
UCXvKUavCtDOlA8bT1i2tI3w is 22 chars after UC and match description
Missing search log in logs so cannot check guide assignment but missing token ID file...
Sorry and thanks for your time. Here's both logs after:
- Adding
X-Plex-Token.idgot from here to this path~\AppData\Local\Plex Media Server\Logs - Deleting all Plex logs leaving folders intact
- Restarting Plex
- Updating all metadata for the YouTube library
It still isn't working unfortunately…
com.plexapp.agents.youtube.log com.plexapp.system.log
Language xn for library = Chinese?
Language is set to None in Library settings, unsure why the logs state "xn"
If it's any help I'm using yt-dlp 2021.12.27.
Library file tree
.
├── belle delphine [UCXvKUavCtDOlA8bT1i2tI3w]
│ ├── 2020-06-17 - I'M BACK -belle delphine [TL470fJMi7w].description
│ ├── 2020-06-17 - I'M BACK -belle delphine [TL470fJMi7w].info.json
│ ├── 2020-06-17 - I'M BACK -belle delphine [TL470fJMi7w].mkv
│ ├── 2020-06-17 - I'M BACK -belle delphine [TL470fJMi7w].webp
│ ├── 2021-02-14 - my room tour (worlds largest sex toy collection idk) [cfLX_s8bsfE].description
│ ├── 2021-02-14 - my room tour (worlds largest sex toy collection idk) [cfLX_s8bsfE].info.json
│ ├── 2021-02-14 - my room tour (worlds largest sex toy collection idk) [cfLX_s8bsfE].mkv
│ ├── 2021-02-14 - my room tour (worlds largest sex toy collection idk) [cfLX_s8bsfE].webp
│ ├── NA - belle delphine - Videos [UCXvKUavCtDOlA8bT1i2tI3w].description
│ ├── NA - belle delphine - Videos [UCXvKUavCtDOlA8bT1i2tI3w].info.json
│ └── NA - belle delphine - Videos [UCXvKUavCtDOlA8bT1i2tI3w].jpg
└── yotobi [UCIPsK5xspHC3-ZFNPTx2X_w]
├── 2007-02-12 - Sad moment from scrubs [Lv36agD7Tfo].description
├── 2007-02-12 - Sad moment from scrubs [Lv36agD7Tfo].info.json
├── 2007-02-12 - Sad moment from scrubs [Lv36agD7Tfo].mkv
└── 2007-02-12 - Sad moment from scrubs [Lv36agD7Tfo].webp
I tried an old version of YouTube-Agent.bundle I had on an iMac. That one downloads metadata and images for the channels but with wrong image and it sets the title to the filepath.
This old version: YouTube-Agent.bundle.zip
Can access library file now but issue persists. No search was done on that series in logs...
I'm sorry I don't understand if I should search something on Plex or you're referring to the agent not searching?
I need you to match manually the series so I have that in logs, the agent search() funct89n assign a unique I'd and it seems it is what doesn't work here
Hello again, hope you have a good week.
If you meant "Fix match" and then manually matching the YouTube channel, I did just that and the correct metadata for the channel got scraped BUT now the episodes metadata don't get collected.
Here are the logs:
com.plexapp.agents.youtube.log com.plexapp.system.log
Just wanted to chime in that I'm also seeing this issue - episode metadata is loading fine but series metadata no longer is. No images for the series and the [RANDOM LETTERS AND NUMBERS] are still in the title of the series. Let me know if you'd like me to get logs also.
Just wanted to chime in that I'm also seeing this issue - episode metadata is loading fine but series metadata no longer is. No images for the series and the [RANDOM LETTERS AND NUMBERS] are still in the title of the series. Let me know if you'd like me to get logs also.
@cbartondock please do! That way ZeroQI can see your logs together with mine as soon as they have time. I'm pasting the instructions to attach the logs if it's of any help:
- Update to the latest Absolute Series Scanner, Youtube-Agent
- Delete all Plex logs leaving folders intact
- Restart Plex
- Update the series Metadata
- Include all the following logs: (location)
-
[...]/Plex Media Server/Logs/PMS Plugin Logs/com.plexapp.agents.Youtube-Agent.log(Agent logs) -
[...]/Plex Media Server/Logs/PMS Plugin Logs/com.plexapp.system.log(show why the agent cannot launch)
-
- Attach some screenshots to illustrate if helpful.
Same error as you looks like, the key line being Error opening URL 'http://127.0.0.1:32400/:/plugins/com.plexapp.system/resourceHashes'.
com.plexapp.agents.youtube.log com.plexapp.system.log
@mileyvirus Does the X-Plex-Token.id file just have the token string and nothing else in it? Doesn't seem to be working for me.
@cbartondock yes, just the token string and nothing else and no line breaks either. I put mine in two places since I didn't figure which was the right path.
C:\Users\<username>\AppData\Local\Plex Media Server\X-Plex-Token.id
C:\Users\<username>\AppData\Local\Plex Media Server\Logs\X-Plex-Token.id
The content of X-Plex-Token.id looks like this (I anonymized mine for posting it here)
HEYXX9aT8UxW7igoDuCk
Okay yeah, I just worked out by looking over __index.py__ that actually the correct path is the first one C:\Users\Chris Barton Dock\AppData\Local\Plex Media Server. Now my logs look better, but still no dice. Going to try debugging __index.py__ further.
com.plexapp.agents.youtube.log
com.plexapp.system.log
.
It seems to be grabbing the guid from a random episode of the channel rather than for the channel itself.
"No GUID so random folder" so it considers the folder doesn't have a channel ID indicated... You should output the metadata.id in logs to see if the search part got the channel id, and if not, you might have used the Plex scanner that would delete all in square brackets which is the channel id...
Pretty sure this is a bug. It is setting the channel guid to the guid of a random episode. metadata.id is youtube|aPhrTOg1RUk|Contrapoints [UCNvsIonJdJ5E4EXMa65VYpA] when I believe it should be youtube|UCNvsIonJdJ5E4EXMa65VYpA|Contrapoints [UCNvsIonJdJ5E4EXMa65VYpA]. Here is the directory
And here is the Info.xml file that is being set in C:\Users\Chris Barton Dock\AppData\Local\Plex Media Server\Metadata\TV Shows\e\73abc82969112e63ba6300258b3a9a97c05c7a2.bundle\Contents\com.plexapp.agents.youtube:
I believe this is occurring as early as media.filename in the Search(...) function.
The reason it says No GUID is because the GUID is an episode GUID, which it doesn't recognize since it doesn't start with UC or HC
ASS scanner let the title with square brackets with channel I'd pass and the agent search function assign a forced guid. Plex scanner will drop square brackets and would cause that exact behaviour, so is ASS the scanner? If so, un-assign, re-assign the series and attach the agents logs so they include the search part.
Edit: Please test https://github.com/ZeroQI/YouTube-Agent.bundle/blob/master/Contents/Code/init.py line 120 should be display_name instead of filename
Ah I see, I don't think I am using ASS and I see now that it is a requirement for YouTube-Agent. Weird that it was working before. If you don't mind my asking, the documentation for ASS (what it does and how to use it for YouTube-Agent) is pretty opaque, how do I set-up ASS?
If not no worries, I got it working in the end without ASS by patching __init__.py with the following (added after line 109):
a_list=[x for x in os.listdir(dir) if x.endswith('.info.json') and re.search(r'\[(.*)\]',x).group(1)[0:2] in ('PL', 'UU', 'FL', 'LP', 'RD','UC','HC')]
if len(a_list)>0:
good_guid = re.search(r'\[(.*)\]',a_list[0]).group(1)
try: json_video_details = JSON.ObjectFromString(Core.storage.load(json_filename)) #with open(json_filename) as f: json_video_details = JSON.ObjectFromString(f.read())
except: pass
else:
results.Append( MetadataSearchResult( id='youtube|{}|{}'.format(good_guid, os.path.basename(dir)), name=displayname, year=Datetime.ParseDate(Dict(json_video_details, 'upload_date')).year, score=100, lang=lang ) )
Log(u''.ljust(157, '='))
return
Basically it just forcibly gets the good guid from the correct .info.json
Edit: Please test https://github.com/ZeroQI/YouTube-Agent.bundle/blob/master/Contents/Code/init.py line 120 should be display_name instead of filename
@ZeroQI I tried that change but it still won't work for me. I have the latest ASS.
If not no worries, I got it working in the end without ASS by patching init.py with the following (added after line 109):
@cbartondock I tried your change but it won't work for me.
I'm a Python 3 dev but opening the Agent's code it looks like it's all Python 2 and it has limited libraries. I'm going to look more into how Plex agents work. I tried the other YouTube agent and that works but with some other issues…
ASS is a requirement for playlist and channel guide in title functionality, as well as numbering playlist correctly...
My agent also read metadata from local Json file if present like the other agent...
Where are the search agent logs after the change? The change will take the ass title with brackets, will make no change with Plex default scanner...
We need code working within Json then working with local Json file so it benefits everybody. I unfortunately don't use Plex anymore, so don't reproduce libraries, and so need the log to do the coding on the phone most of times
I was able to recreate this issue and resolve it ( at least for myself). Initially I was using the TV scanner, and I inserted a "air date" into the file name to make that scanner happy. I also configured youtube-dl to create an "info.json" file for each episode.
The fix was to change the scanner to the ASS scanner, and remove the "info.json" files from the series folder.
To verify the .json file was the issue, I recreated the library with 2 "series", one with the info.json files and one without. The series without the json file has the correct series title, the other has "foldername + channel id" in the name.
One think I could not do was remove the json file and rescan the library to correct the series name. Perhaps I am being impatient, or I'm not performing the correct steps to fix the series name. Ideas?
Without Jason it works With Jason it gives "foldername + channel id" in the name. When you remove the Json file and rescan (after Plex dance), does it changes anything? Would need agent logs for both situations
I can't attach files here. I'll open another issue. with logs and additional details.
edit 1: it looks like github doesn't let you attach logs to issues. I'm looking for an alternative way to share the logs.
edit 2: it accepts .log but not archive file like .tgz. I'll post logs and details in another post.
Issue
Channel metadata inc poster missing. episode name/metadata is correct. Channel name is the channel folder name.
*Example: *
Folder is Don_t_Walk_Run_Productions [UCwpDBy43upJR7LZLrgnKkPA], and channel name is Don_t_Walk_Run_Productions [UCwpDBy43upJR7LZLrgnKkPA].
Notable Observations
- The ASS scanner *.hama directory does not exist.
- ASS Scanner logs are empty.
- Error decoding with simplejson warning.
- The resourceHashes critical error.
- The cause for the "No GUID" is probably what I need to solve.
- I haven't looked at the Agent code, so I don't know the logic it uses to determine the Channel ID.
Environment
- OS: Debian v11.3
- Plex Media Server: v1.26.2.5797
- ASS Version: commit id 11c303b.
- Youtube Agent: I don't have the version. But I downloaded and installed it on "Apr 25 06:02"
Plex Token ID File: /var/lib/plexmediaserver/Library/Application Support/Plex Media Server/X-Plex-Token.id
Log Excerpts
Plex token and json from top part of com.plex.agent.youtube.log:
2022-05-20 14:06:57,086 (7f162ad1c808) : INFO (__init__:549) - 'X-Plex-Token.id' file present
2022-05-20 14:06:57,100 (7f1626e3bb38) : WARNING (data:179) - Error decoding with simplejson, using demjson instead (this will cause a performance hit) - Expecting value: line 5 column 1 (char 518)
Exception getting hosted resource hashes from com.plex.agent.youtube.log:
2022-05-20 12:53:17,364 (7fc7e8ad8808) : DEBUG (networking:143) - Requesting 'http://127.0.0.1:32400/:/plugins/com.plexapp.system/resourceHashes'
2022-05-20 12:53:17,367 (7fc7e8ad8808) : ERROR (networking:196) - Error opening URL 'http://127.0.0.1:32400/:/plugins/com.plexapp.system/resourceHashes'
2022-05-20 12:53:17,369 (7fc7e8ad8808) : CRITICAL (runtime:1299) - Exception getting hosted resource hashes (most recent call last):
File "/usr/lib/plexmediaserver/Resources/Plug-ins-5bd057d2b/Framework.bundle/Contents/Resources/Versions/2/Python/Framework/components/runtime.py", line 1293, in get_resource_hashes
json = self._core.networking.http_request("http://127.0.0.1:32400/:/plugins/com.plexapp.system/resourceHashes", timeout=10).content
File "/usr/lib/plexmediaserver/Resources/Plug-ins-5bd057d2b/Framework.bundle/Contents/Resources/Versions/2/Python/Framework/components/networking.py", line 242, in content
return self.__str__()
File "/usr/lib/plexmediaserver/Resources/Plug-ins-5bd057d2b/Framework.bundle/Contents/Resources/Versions/2/Python/Framework/components/networking.py", line 220, in __str__
self.load()
File "/usr/lib/plexmediaserver/Resources/Plug-ins-5bd057d2b/Framework.bundle/Contents/Resources/Versions/2/Python/Framework/components/networking.py", line 158, in load
f = self._opener.open(req, timeout=self._timeout)
File "/usr/lib/plexmediaserver/Resources/Python/python27.zip/urllib2.py", line 435, in open
response = meth(req, response)
File "/usr/lib/plexmediaserver/Resources/Python/python27.zip/urllib2.py", line 548, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib/plexmediaserver/Resources/Python/python27.zip/urllib2.py", line 473, in error
return self._call_chain(*args)
File "/usr/lib/plexmediaserver/Resources/Python/python27.zip/urllib2.py", line 407, in _call_chain
result = func(*args)
File "/usr/lib/plexmediaserver/Resources/Python/python27.zip/urllib2.py", line 556, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 404: Not Found
Data for single channel from com.plex.agent.youtube.log:
2022-05-20 14:10:27,045 (7f1626e3bb38) : DEBUG (networking:143) - Requesting 'http://127.0.0.1:32400/library/metadata/226165/tree'
2022-05-20 14:10:27,053 (7f1626e3bb38) : INFO (__init__:165) - === update(lang=xn, force=True, movie=False) ===
2022-05-20 14:10:27,053 (7f1626e3bb38) : INFO (__init__:40) - /mnt/smb/youtube2/Don_t_Walk_Run_Productions [UCwpDBy43upJR7LZLrgnKkPA]/The_Press_vs_Trump_-_Volume_Three_The_Charlottesville_Lie [4jTf3CMbHrk].mp4
2022-05-20 14:10:27,054 (7f1626e3bb38) : INFO (__init__:509) - =============================================================================================================================================================
2022-05-20 14:10:27,054 (7f1626e3bb38) : INFO (__init__:252) - [ ] library: "YT"
2022-05-20 14:10:27,054 (7f1626e3bb38) : INFO (__init__:253) - [ ] root: "/mnt/smb/youtube2"
2022-05-20 14:10:27,054 (7f1626e3bb38) : INFO (__init__:254) - [ ] path: "Don_t_Walk_Run_Productions [UCwpDBy43upJR7LZLrgnKkPA]"
2022-05-20 14:10:27,054 (7f1626e3bb38) : INFO (__init__:255) - [ ] dir: "/mnt/smb/youtube2/Don_t_Walk_Run_Productions [UCwpDBy43upJR7LZLrgnKkPA]"
2022-05-20 14:10:27,054 (7f1626e3bb38) : INFO (__init__:260) - [ ] series_root_folder: "/mnt/smb/youtube2/Don_t_Walk_Run_Productions [UCwpDBy43upJR7LZLrgnKkPA]"
2022-05-20 14:10:27,056 (7f1626e3bb38) : INFO (__init__:263) - [ ] subfolder_count: "0"
2022-05-20 14:10:27,057 (7f1626e3bb38) : INFO (__init__:378) - Grouping folder not found or single folder, root: /mnt/smb/youtube2, path: Don_t_Walk_Run_Productions [UCwpDBy43upJR7LZLrgnKkPA], Grouping folder: Don_t_Walk_Run_Productions [UCwpDBy43upJR7LZLrgnKkPA], subdirs: 0, reverse_path: [u'Don_t_Walk_Run_Productions [UCwpDBy43upJR7LZLrgnKkPA]']
2022-05-20 14:10:27,057 (7f1626e3bb38) : INFO (__init__:400) - No GUID so random folder
2022-05-20 14:10:27,057 (7f1626e3bb38) : INFO (__init__:410) - =============================================================================================================================================================
Log file attachments
com.plexapp.agents.youtube.log
ASS Scanner Logs for channel I'm focusing on were empty so I can't attach them. Here are their names.
- Don_t_Walk_Run_Productions [UCwpDBy43upJR7LZLrgnKkPA].filelist.log
- Don_t_Walk_Run_Productions [UCwpDBy43upJR7LZLrgnKkPA].scanner.log
ok.I redownloaded my library again with better results. Since I started, I made the following changes
- I added the Plex Token file
- (yt-dlp) added --restrict-filename to only allow ascii filenames.
- (yt-dlp) included the channel name in the filename to remove non-ascii chars from that as well. My channel name had some accented chars in them.
Channel Names , episode names, metadata, and images are all correct now. ( yay!) .
However, I found a new problem. Youtube episodes released on the same day are absent. I had a case where 5 of them had the same modified date and none of them were discovered by the scanner., The Plex scanner log has the write number of episodes, but doesn't add them.
This is my file naming convention, is correct? Or should I change it? I don't use playlist so there isn't a playlist id.
-o "%(channel)s [%(channel_id)s]/%(title)s [%(id)s].%(ext)s"
The overall script looks like this
${ytdlp_path} \ --match-filter "is_live != true & was_live != true" \ ${PROXY_CMD} \ --abort-on-error \ --no-warnings \ --restrict-filenames \ --no-progress \ --add-metadata \ --ffmpeg-location "${ffmpeg_path}" \ --recode "mp4" \ --batch-file="${batch_file}" \ --cookies "${cookies_file}" \ --download-archive "${download_archive}" \ -o "%(channel)s [%(channel_id)s]/%(title)s [%(id)s].%(ext)s" \ --exec post_process:"./scripts/move_video_file.sh -f '%(filepath)s' -r '${root_path}'"
I made the mistake of touching all of 24 file in a channel folder, and not only one is shown in Plex. So the scanner seems to be using the modified date and only allowing one episode per day.
I tried a few things, but what got me results was adding MMDDhhmm to the file name. The scanner picks up these files and I see them in Plex. But the youtube agent doesn't fetch the metadata then, so the episode title /etc are wrong. The filename becomes the title.
So I need a file naming format that makes both ASS and Youtube Agent happy.
I resolved the episode issue with 2 changes
- change the youtube type to "youtube2" by changing the channel ID field to [youtube2-id] as per the documentation.
- (tricky part) I scripted adding a randomized hour and minute to the "release_date" value from yt-dlp. The value is applied to the video file after copying to the destination folder.
# generate random hour/minute
HH=$(( 1 + $RANDOM % 23))
MM=$((1 + $RANDOM % 60))
# change modified date using touch command
touch -m -t "${date_string}${HH}${MM}" "${DESTPATH}${FILENAME}"
ASS will process all of the video files include those with identical release dates.
I'm creating a dev environment to see if I come up with a way to have ASS handle identical release dates on it's own. Moving responsibility for this from the end user to the application.
Btw, ASS/Youtube-Agent are great tools. I threw a donation the dev's way and recommend others do the same.
I believe the root causes for the issue I ran into were the missing Plex token, the non-ascii characters, and duplicate release_dates on video files in the same series.