omnivore icon indicating copy to clipboard operation
omnivore copied to clipboard

Bug: There was an error adding new feed: Item not found

Open flethj opened this issue 11 months ago • 19 comments

I'm running into an issue when trying to import an RSS feed.

Feed: https://openrss.org/https://www.youtube.com/@ByteByteGo/videos

Error: "There was an error adding new feed: Item not found"

The feed appears to exist and it can be imported into RSS readers (I tried Reeder and it worked). However, it does not work in Omnivore. I tried both the web and iOS versions.

flethj avatar Mar 14 '24 06:03 flethj

Hey @flethj are you directly adding that URL? It looks like an HTML page to me but maybe its meant to change based on accepted content type?

jacksonh avatar Mar 14 '24 06:03 jacksonh

Hello, I tried to add exactly this url. The returned content probably depends on something like the accepted content type. Chrome shows the HTML page but for example Safari recognizes the RSS feed and asks whether to open it in an RSS reader.

The weird thing is that I've successfully used openrss.org in combination with YouTube channels in Omnivore before. Exactly as above but with a different channel. A different RSS client is also (still) able to see the feed with this exact url, which is why I suspect that something in Omnivore broke.

flethj avatar Mar 14 '24 09:03 flethj

Yeah its likely because we added some extra accepted content types because a few other feeds required things like HTML be accepted, probably due to misconfigured load balancers.

jacksonh avatar Mar 14 '24 09:03 jacksonh

I get the same error adding this feed:

https://direct.mit.edu/rss/site_1000093/1000049.xml

I guess the problem has to do with the presence of an underscore in the link. Should I open a new issue?

niksart avatar Mar 17 '24 10:03 niksart

I get the same error adding this feed:

https://direct.mit.edu/rss/site_1000093/1000049.xml

I guess the problem has to do with the presence of an underscore in the link. Should I open a new issue?

curl https://direct.mit.edu/rss/site_1000093/1000049.xml 
<html><body><h1>403 Forbidden</h1>
Request forbidden by administrative rules.
</body></html>

looks like they have some blocking of that feed.

jacksonh avatar Mar 18 '24 01:03 jacksonh

It seems that the problem is the user agent "curl" that is blocked by them:

curl -A "curl" "https://direct.mit.edu/rss/site_1000093/1000049.xml"
<html><body><h1>403 Forbidden</h1>
Request forbidden by administrative rules.
</body></html>

Any other string works. Try for example:

curl -A "qwertyuiop" "https://direct.mit.edu/rss/site_1000093/1000049.xml"

niksart avatar Mar 18 '24 21:03 niksart

Yeah i suspect they are also blocking some IPs, i can add this feed fine in local development but can't from our backend.

jacksonh avatar Mar 19 '24 08:03 jacksonh

Hello, are there any plans to change the accepted content types again (so that openrss.org works again)?

fabianlandwehr1 avatar Apr 11 '24 06:04 fabianlandwehr1

Same problem for me here:

  • https://www.geektime.co.il/feed/

$ curl https://orikatz.wordpress.com/feed/

<?xml version="1.0" encoding="UTF-8"?><rss version="2.0" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:wfw="http://wellformedweb.org/CommentAPI/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:sy="http://purl.org/rss/1.0/modules/syndication/" xmlns:slash="http://purl.org/rss/1.0/modules/slash/" xmlns:georss="http://www.georss.org/georss" xmlns:geo="http://www.w3.org/2003/01/geo/wgs84_pos#" xmlns:media="http://search.yahoo.com/mrss/" > <channel>............

The URL works in feedly.

CopyPasteFail avatar Apr 28 '24 07:04 CopyPasteFail

I'm facing the same issue with the DistroWatch RSS feeds.

micmalti avatar Jun 04 '24 16:06 micmalti

Same problem for me here:

* https://www.geektime.co.il/feed/

$ curl https://orikatz.wordpress.com/feed/

<?xml version="1.0" encoding="UTF-8"?><rss version="2.0" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:wfw="http://wellformedweb.org/CommentAPI/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:sy="http://purl.org/rss/1.0/modules/syndication/" xmlns:slash="http://purl.org/rss/1.0/modules/slash/" xmlns:georss="http://www.georss.org/georss" xmlns:geo="http://www.w3.org/2003/01/geo/wgs84_pos#" xmlns:media="http://search.yahoo.com/mrss/" > <channel>............

The URL works in feedly.

I get this: https://validator.w3.org/feed/check.cgi?url=https%3A%2F%2Fwww.geektime.co.il%2Ffeed%2F

jacksonh avatar Jun 05 '24 06:06 jacksonh

I'm facing the same issue with the DistroWatch RSS feeds.

Can you give an example URL you are using?

jacksonh avatar Jun 05 '24 06:06 jacksonh

Sure. Here's one: https://distrowatch.com/news/dw.xml

micmalti avatar Jun 05 '24 12:06 micmalti

Same issue with https://www.science.org/digital-feed/pipeline

stevenrobertson avatar Jun 25 '24 16:06 stevenrobertson

Same issue with https://www.inmediahk.net/full/feed

makamto avatar Jul 11 '24 07:07 makamto

Same issue with https://www.rfa.org/cantonese/news/rss2.xml

makamto avatar Jul 18 '24 03:07 makamto

I'm running into an issue when trying to import an RSS feed.

Feed: https://openrss.org/https://www.youtube.com/@ByteByteGo/videos

Error: "There was an error adding new feed: Item not found"

The feed appears to exist and it can be imported into RSS readers (I tried Reeder and it worked). However, it does not work in Omnivore. I tried both the web and iOS versions.

Hey, this seems to be working now

sywhb avatar Jul 18 '24 07:07 sywhb

https://distrowatch.com/news/dw.xml

This seems to be working now

sywhb avatar Jul 18 '24 08:07 sywhb

Hey @jacksonh , I have tested these failed RSS feeds and I get 403 for all of them. Even when I open https://www.inmediahk.net/full/feed in the browser, I sometimes have to click on the Cloudflare captcha which makes me wonder if we should scrape the feed content in puppeteer.

It might be a little helpful but still need to figure out a better way to bypass the captcha with proxy and fingerprint

sywhb avatar Jul 18 '24 09:07 sywhb