not snatching episodes since 2022.07.20

[athen66]

SickChill is not finding any episodes when searching against NZBHydra via API. I can go into NZBHydra manually, see the API search returned results and download manually. Currently running on Ubuntu/Python 3.9 but I've encountered same issues on MacOS and Python 3.7, 3.8 and 3.9.

Version info:

Branch: master Commit: d3d9e0cd9ffe9343563ccbecd5a6c5a55878d0d5 Version: 2022.07.20

This is the error now, that is likely related to the issue, when performing searches against NZBHydra:

Performing episode search for ******************* /opt/SickChill/.venv/lib/python3.9/site-packages/bs4/builder/init.py:545: XMLParsedAsHTMLWarning: It looks like you're parsing an XML document using an HTML parser. If this really is an HTML document (maybe it's XHTML?), you can ignore or filter this warning. If it's XML, you should know that using an XML parser will be more reliable. To parse this document as XML, make sure you have the lxml package installed, and pass the keyword argument features="xml" into the BeautifulSoup constructor. warnings.warn(

[github-actions[bot]]

Hi, thanks for the report. Please use search to make sure your issue has not been reported yet, and someone will try to help you as soon as possible.

[athen66]

screenshot of nzbhydra logs - search is executing fine, so certainly seems like a parsing issue on the sickchill side

[BKSteve]

Believe similar to #8047

The nzb seems to not be updating/parsing.

[DravenSA]

Thought it was just me, I use NZBPlanet, i have not snatched anything in 24 days, Everything has been through Torrents, Not sure where to paste the Debug Log, but it basically says it cant be found, yet i can see it on the site

[miigotu]

Has the new release today fixed this issue?

[Sandrus]

Nothing has been snatched until the 20th of July also, but i don't see any error. Manual snatch is always empty "No cached results, try an automated search first" I use Jackett with a Tornzab feed

[brianjmurrell]

Believe similar to #8047

Indeed. The report in #8047 is using NZBHydra also.

[tking2]

I think I've spotted the issue here, at least in a case I've been facing.

As part of commit db6b1d1ab2dd2f26ac34c146de13de655d8b4032, a new newznab provider has been overriden here: sickchill/oldbeard/providers/newnewznab.py.

Haven't spent much time as to the changes of this file and whethernewznab.py and newnewsznab.py should coexist, but the BS4 parser now uses "xml" as the language here: https://github.com/SickChill/SickChill/blob/master/sickchill/oldbeard/providers/newnewznab.py#L150

The xml language is case sensitive which means certain torznab XML properties will no longer match. My problem stems from being unable to identify the supported params for the provider (passed as supportedparams not supportedParams), resulting in the default params thus invalid search being made.This is found here: https://github.com/SickChill/SickChill/blob/master/sickchill/oldbeard/providers/newnewznab.py#L124

[tking2]

The PR might not fix things if there are other changes as a result of the xml language setting, but hopefully those more familiar with the code will be able to support.

[brianjmurrell]

I'm on latest master: cc9de09d40de432d8d86cd896ec424ab6af0d78d and as I understand, #8087 has been merged into that branch, and am still seeing this issue.

Clearly #8087 is not the cause of this issue.

[tking2]

@brianjmurrell do you have any debug logs you can share when it attempts to search for an episode?

The PR did fix my use case and all is working for me, so I assume this is an alternate issue

[BKSteve]

@brianjmurrell whilst you get some debug logs can you add OS and install type? If you are a pip install have you recently done pip3 install -U sickchill?

[miigotu]

I'm on latest master: cc9de09d40de432d8d86cd896ec424ab6af0d78d and as I understand, #8087 has been merged into that branch, and am still seeing this issue.

Clearly #8087 is not the cause of this issue.

If you look at #8087 it clearly only shows develop in all of the GitHub notifications in the entire thread there. The release will happen as soon as the other providers PR is done and some tests are done. We try to avoid spamming releases before they are tested, because when we do shit breaks more than it does when we test it.

Everything in master branch is a release. We don't merge anything to master between releases unless it is a catastrophic issue.

[brianjmurrell]

Does this not mean this is on master?

[brianjmurrell]

Which I appear to have in my working tree:

[miigotu]

No, because that is not the only fix. You could be helpful and test the develop branch and see if it works, so we know if we can release it or not. Enable debug and make sure to keep logs and explain everything as best you can that is broken.

The reason broken shit ends up on master (in releases) is because not many people will test develop, or even run develop all the time (that's what I do). We need more testers and people willing to contribute documentation or money to pay someone to do documentation/wikis/tests/packages. Testing and contributing with code and docs are so valuable to the project, and the 2 of us working on the code simply cannot keep up with all of it ourselves.

If anyone wants to contribute in an ongoing basis and making commits and edits to wikis I can get them a "all products pack" from intellij, which is worth thousands of dollars ($289/year starting Oct 1) and they can use the software for tons of different coding and scripting languages. It's literally the best coding software available.

[miigotu]

[brianjmurrell]

So the only diff between develop and master currently are:

$ git lg origin/master...origin/develop 
* 78d1e6ce9 - (origin/develop) Update poetry (python) and yarn (npm) dependencies (8 hours ago) <miigotu>
* 0225247fa - Bump xo from 0.52.1 to 0.52.2 (#8115) (24 hours ago) <dependabot[bot]>
* ee6ff0ac5 - Update poetry (python) and yarn (npm) dependencies (32 hours ago) <miigotu>
* 5641e86c7 - Bump xo from 0.52.0 to 0.52.1 (#8112) (2 days ago) <dependabot[bot]>
* 5c3c77435 - Bump xo from 0.51.0 to 0.52.0 (#8110) (2 days ago) <dependabot[bot]>
* d47e51539 - Bump eslint-config-xo from 0.41.0 to 0.42.0 (#8111) (2 days ago) <dependabot[bot]>
* 59007db63 - Update poetry (python) and yarn (npm) dependencies (2 days ago) <miigotu>
* 87865e418 - Update poetry (python) and yarn (npm) dependencies (3 days ago) <miigotu>
* 333f6f65a - Update poetry (python) and yarn (npm) dependencies (4 days ago) <miigotu>

Doesn't seem like a lot of difference to me. Certainly the commit above that is supposed to fix this is not in the diff so it must be on master already, yes?

[brianjmurrell]

So I have been digging into this a bit more. I am not a SC hacker by any means, but I do know how to read and write python. I am hoping those with better overall understanding can help me fill the gaps.

My understanding is that ultimately, there is a results table in cache.db that is searched by TVCache.find_needed_episodes() for the case of a newznab provider for an RSS search. Hopefully I am correct so far.

In my cache.db that table is empty however. But I cannot seem to figure out the code path that populates it, from an RSS fetch. Any pointers? I'll keep digging in the meanwhile.

[brianjmurrell]

No, because that is not the only fix. You could be helpful and test the develop branch and see if it works,

As I suspected, it does not work on develop either.

Here is the result of doing a manual search for an episode that is Wanted, with debug logging enabled:

2022-08-29 10:31:39 DEBUG :: SEARCHQUEUE-MANUAL-1234 :: formatting pattern: %SN - S%0SE%0E - %EN -> Show Name - S01E02 - Episode Name
2022-08-29 10:31:39 INFO :: SEARCHQUEUE-MANUAL-1234 :: Beginning manual search for: [Show Name - S01E02 - Episode Name]
2022-08-29 10:31:40 DEBUG :: SEARCHQUEUE-MANUAL-1234 :: Building internal name cache for Show Name
2022-08-29 10:31:40 DEBUG :: SEARCHQUEUE-MANUAL-1234 :: get_all_scene_exceptions: {-1: [{'show_name': 'show name', 'custom': False}]}
2022-08-29 10:31:40 DEBUG :: SEARCHQUEUE-MANUAL-1234 :: exceptions_cache for 1234: {-1: ['show name']}
2022-08-29 10:31:40 DEBUG :: SEARCHQUEUE-MANUAL-1234 :: Internal name cache for Show Name set to: [ show name ]
2022-08-29 10:31:40 DEBUG :: SEARCHQUEUE-MANUAL-1234 :: [My NZBHydra 2] :: Last update was too soon, using old cache: 2022-08-29 10:16:16. Updated less then 30 minutes ago
2022-08-29 10:31:40 INFO :: SEARCHQUEUE-MANUAL-1234 :: [My NZBHydra 2] :: Performing episode search for Show Name
2022-08-29 10:31:42 DEBUG :: SEARCHQUEUE-MANUAL-1234 :: [My NZBHydra 2] :: get_scene_exceptions: ['show name']
2022-08-29 10:31:42 DEBUG :: SEARCHQUEUE-MANUAL-1234 :: [My NZBHydra 2] :: exceptions_cache: {-1: ['show name']}
2022-08-29 10:31:42 DEBUG :: SEARCHQUEUE-MANUAL-1234 :: [My NZBHydra 2] :: get_scene_exceptions: ['show name']
2022-08-29 10:31:43 DEBUG :: SEARCHQUEUE-MANUAL-1234 :: [My NZBHydra 2] :: exceptions_cache: {-1: ['show name']}
2022-08-29 10:31:43 DEBUG :: SEARCHQUEUE-MANUAL-1234 :: [My NZBHydra 2] :: Search Mode: Episode
2022-08-29 10:31:43 DEBUG :: SEARCHQUEUE-MANUAL-1234 :: [My NZBHydra 2] :: Search String: show name S01E02
2022-08-29 10:31:46 DEBUG :: SEARCHQUEUE-MANUAL-1234 :: [My NZBHydra 2] :: GET URL: http://nzb:5076/api?t=tvsearch&limit=100&offset=0&cat=5000%2C5040%2C5030&maxage=500&apikey=[redacted]&tvdbid=1234&season=1&ep=2 [Status: 200]
2022-08-29 10:31:46 DEBUG :: SEARCHQUEUE-MANUAL-1234 :: formatting pattern: %SN - S%0SE%0E - %EN -> Show Name - S01E02 - Episode Name
2022-08-29 10:31:46 DEBUG :: SEARCHQUEUE-MANUAL-1234 :: formatting pattern: %SN - S%0SE%0E - %EN -> Show Name - S01E02 - Episode Name
2022-08-29 10:31:46 INFO :: SEARCHQUEUE-MANUAL-1234 :: Unable to find a download for: [Show Name - S01E02 - Episode Name]

And as before, I can confirm that the NZB result does have what should match:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<rss xmlns:atom="http://www.w3.org/2005/Atom"  xmlns:newznab="http://www.newznab.com/DTD/2010/feeds/attributes/" version=
"2.0">
    
    <channel>
        <title>NZBHydra 2</title>
        <link>https://www.github.com/theotherp/nzbhydra2</link>
        <webMaster>[email protected]</webMaster>
        <generator>NZBHydra2</generator>
        <newznab:response offset="0" total="33"/>
	...
        <item>
            <title>Show.Name.2022.S01E02.Episode.Name.1080p</title>
            <link>http://nzb:5076/getnzb/api/-7301010754180609597?apikey=[redacted]</link>
            <enclosure url="http://nzb:5076/getnzb/api/-7301010754180609597?apikey=[redacted]" length="1583166513" type="application/x-nzb"/>
            <pubDate>Mon, 29 Aug 2022 04:56:02 +0000</pubDate>
            <guid isPermaLink="false">-7301010754180609597</guid>
            <description>Show.Name.2022.S01E02.Episode.Name.1080p.HMAX.Webrip.x265.10bit.EAC3.5.1.Atmos.GokiTAoE</description>
            <comments>https://drunkenslug.com/details/[redacted]#comments</comments>
            <category>TV HD</category>
            <size>1583166513</size>
            <newznab:attr name="category" value="5040"/>
            <newznab:attr name="comments" value="0"/>
            <newznab:attr name="episode" value="2"/>
            <newznab:attr name="files" value="19"/>
            <newznab:attr name="grabs" value="794"/>
            <newznab:attr name="group" value="alt.binaries.coolkidweb"/>
            <newznab:attr name="guid" value="-7301010754180609597"/>
            <newznab:attr name="imdbid" value="11198330"/>
            <newznab:attr name="password" value="0"/>
            <newznab:attr name="poster" value="[email protected]"/>
            <newznab:attr name="season" value="1"/>
            <newznab:attr name="showtitle" value="showname2022"/>
            <newznab:attr name="size" value="1583166513"/>
            <newznab:attr name="title" value="Episode Name"/>
            <newznab:attr name="tmdbid" value="[redacted]"/>
            <newznab:attr name="traktid" value="[redacted]"/>
            <newznab:attr name="tvairdate" value="2022-08-28"/>
            <newznab:attr name="tvdbid" value="[redacted]"/>
            <newznab:attr name="tvmazeid" value="[redacted]"/>
            <newznab:attr name="usenetdate" value="Mon, 29 Aug 2022 05:51:59 +0100"/>
            <newznab:attr name="hydraIndexerScore" value="0"/>
            <newznab:attr name="hydraIndexerHost" value="api.drunkenslug.com"/>
            <newznab:attr name="hydraIndexerName" value="Drunken Slug 1"/>
        </item>
	...
    </channel>
</rss>

[brianjmurrell]

Issue discovered. #8047 is not a duplicate of this issue.

[miigotu]

Issue discovered. #8047 is not a duplicate of this issue.

https://github.com/SickChill/SickChill/issues/8047#issuecomment-1230610172