rssbox
rssbox copied to clipboard
stefansundin
Instagram discussion issue
It's not fixable. They are rate limiting much harder now. If everyone would just stop hammering my free service everyone would just have a better experience.
The only good fix is to host your own RSS Box. It's not that hard, give it a try.
It's not fixable. They are rate limiting much harder now. If everyone would just stop hammering my free service everyone would just have a better experience.
The only good fix is to host your own RSS Box. It's not that hard, give it a try.
I'm hosting it myself. It has the same problem.
Try adding a sessionid as described here: https://github.com/stefansundin/rssbox/issues/21#issuecomment-525130553
Maybe I should just disable all requests without one.
Same problem here with my own heroku hosted env. Did not try the sessionid thing yet.
I'm also hosting it privately on a dedicated server and getting :
"There was a problem talking to Instagram. Please try again in a moment."
Reading instagram.rb I suspect it has nothing to do with the rate limit , as you have defined custom rate limit error, and this is not the error raised in the output (InstagramError vs InstagramRatelimitError)
Here's the actual instagram response in my case
2020-09-04 15:39:43 - InstagramError - 302: https://www.instagram.com/accounts/login/?next=/web/search/topsearch/%3F__a%3D1%26query%3
Is Instagram refusing all request to anonymous users now ? Why is it redirecting to login page ? To be clear I tried it with a public page URI, I have no problem while using a private browser with no cookie whatsoever
I haven't tried your solution with the sessionid because I think it's not "production ready " to put personal cookie into a public facing webapp. What if I don't want to create an Instagram account at all ?
PS:
The log is truncated as defined in your http.rb:138 : truncate the message in order to cut down on log filesize
If you're interested in reducing the logfile you could look into "system level" solutions like logrotate, in order to keep the integrity of data instead.
Try adding a
sessionidas described here: #21 (comment)
I'm using sessionid but instagram is complaining about suspicious access the last week, I was forced to change my password 3 times in a week already! is rssbox the culprit? did anyone else have similar issues?
Is Instagram refusing all request to anonymous users now ? Why is it redirecting to login page ?
I don't know why they sometimes return 429 and sometimes a redirect to the login page. I suspect there may be two different rate limit protections, perhaps one to just get people to login on the application layer, and then another one at an infrastructure layer that based on IP address that returns 429. This is just a guess though.
I haven't tried your solution with the sessionid because I think it's not "production ready " to put personal cookie into a public facing webapp. What if I don't want to create an Instagram account at all ?
None of this is sanctioned by Instagram so don't ever expect the Instagram feature to be "production ready". lol. And use the sessionid hack at your own risk.
If you're interested in reducing the logfile you could look into "system level" solutions like
logrotate, in order to keep the integrity of data instead.
Not possible on Heroku. That's what I optimize the app for by default. You can change things like log truncation on your own.
I'm using sessionid but instagram is complaining about suspicious access the last week, I was forced to change my password 3 times in a week already! is rssbox the culprit? did anyone else have similar issues?
Probably. Use at your own risk.
IFTTT Activity Said:
Applet failed Feedjira::UserFacingFetchFailure
Show details: No details available.
So I recently implemented some pretty good caching functionality, which has greatly stabilized RSS Box. The app is really responsive now and I am really happy with how well it has restored usability of the website. However, on its own it is not enough to make Instagram work well.
Currently you can only configure one "Instagram sessionid" by setting INSTAGRAM_SESSIONID. And if you configure it and do enough requests quickly enough then Instagram will say that they have detected "unusual activity" from your account and you will have to perform some manual steps to regain access.
I have not added a real phone number yet so I don't know if you can continue making requests afterwards. Please comment here with your experience if you try this. Right now rssbox.herokuapp.com is running with INSTAGRAM_SESSIONID unset.
If I could add code that tracks the number of requests that are being sent to Instagram, then we could try to find a request rate that is safe to use without locking out the account. Then I can add some code that allows a lot of session ids to be added (probably using redis?), and the application can "load balance" between the ids to avoid using one too much. It would take some time to implement though.
I would need help from volunteers to help me register a lot of accounts so we can have a collection of instagram session ids. I don't know how many ids we'd need, but the more the merrier. And it's not possible to sign up multiple accounts using email tricks such as using + at the end of the email, or multiple dots. But maybe there's a trick to work around this that someone knows about?
Anyway, we can continue using this issue to discuss the issue. Let's focus on trying to find a solution. (Any unhelpful comment saying "please fix" or similar will be removed.)
@stefansundin, after I confirmed with my phone number and got back into Instagram, then refreshed Instagram session ID (I don't remember whether it has changed), RSSBox started working.
@ALERTua That's great to hear. How many feeds are you subscribed to using that session id? Let us know if it stops working again, or if you have to take any more actions to verify your account.
@stefansundin ~100 feeds. 12 hours between updates are pulled by my TinyTinyRSS. okay, I will let you know here if anything goes wrong or Instagram wants any verification from me.
@ALERTua How did you get this working? I inputted the INSTAGRAM_SESSIONID, yet I am still getting hit with the rate limiting message in the UI.
@TalD
-
Open new browser tab
-
Press F12 and go to Network tab
-
Go to instagram.com
-
In the developers console Scroll all the way up to the www.instagram.com Name
-
Highlight it. Window splits vertically.
-
At the right side of the vertical split go to Headers tab
-
At the Headers tab scroll down to Request Headers field cookie
-
Within the cookie value find sessionid=49000000%3ABCDEFGHHVABCDE%3AB;
-
Copy this value except 'sessionid=' and the trailing ';'
-
Open your Dockerfile/docker-compose/Vagrantfile/Unraid/whatever else you are using to deploy rssbox
-
Add environment variable INSTAGRAM_SESSIONID and with the copied sessionid value
-
If you have already deployed rssbox - do not forget to recreate the container/vm
-
Use rssbox as before, but now all your instagram.com requests will be made as your user.
update: lowered the refresh rate for Instagram feeds to 4 hours and got my account locked :D
Instagram demanded a password change and everything got working again.
Raised refresh rate back to 12 hours.
got suspended again. this time Instagram demanded phone confirmation and captcha and gave me 30 days to reactivate before banning.
raised feeds update interval to 24 hours
I've given up on the Instagram feeds.
The rate limiting / getting locked out is too unpredictable and annoying.
FYI, there is a Chrome plugin called Feedbro.
It's a bit manual, but after you input / organize your feeds, it works like any RSS feed.
Only downside is you need to be on your desktop.
My account is locked. And the Instagram mobile only unlock said that it is possible and said, shows a 360-degree, and my face. It's terrible.
why not just remove instagram from rssbox.herokuapp.com if you can't fix it? Very sorry. But out of desire to watch a couple of people, in no case do you want to create an account on this garbage dump
I'm still using my rssbox instance to get updates for ~200 Instagram accounts. I don't get many Instagram account verification requests anymore. I guess I proved enough that I'm a real person :D The session id changes after almost every verification, but I'm ok with this. Please leave the Instagram support for rssbox working as it is. Yes, there are service limitations, but it is understandable that Instagram doesn't want crawlers to get data, but a real person watching their advertisement.
I just deployed my own rssbox and (without a sessionid) I'm getting the rate-limited error right away, which is weird.
something broke. rssbox responds correctly with the URL of the feed, but the feed is "Something went wrong. Try again later." Logs:
192.168.1.1 - - [03/Feb/2022:10:42:25 +0200] "GET /instagram?q=username HTTP/1.1" 200 - 2.0286
2022-02-03 10:42:26 - NoMethodError - undefined method `[]' for nil:NilClass:
/app/app/instagram.rb:40:in `block in get_post'
/app/lib/cache.rb:90:in `cache'
/app/app/instagram.rb:36:in `get_post'
/app/app.rb:626:in `block (3 levels) in <top (required)>'
/app/app.rb:624:in `map'
/app/app.rb:624:in `block (2 levels) in <top (required)>'
/app/lib/cache.rb:90:in `cache'
/app/app.rb:614:in `block in <top (required)>'
/app/.bundle/gems/ruby/3.0.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1674:in `call'
/app/.bundle/gems/ruby/3.0.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1674:in `block in compile!'
/app/.bundle/gems/ruby/3.0.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1013:in `block (3 levels) in route!'
/app/.bundle/gems/ruby/3.0.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1032:in `route_eval'
/app/.bundle/gems/ruby/3.0.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1013:in `block (2 levels) in route!'
/app/.bundle/gems/ruby/3.0.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1061:in `block in process_route'
/app/.bundle/gems/ruby/3.0.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1059:in `catch'
/app/.bundle/gems/ruby/3.0.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1059:in `process_route'
/app/.bundle/gems/ruby/3.0.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1011:in `block in route!'
/app/.bundle/gems/ruby/3.0.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1008:in `each'
/app/.bundle/gems/ruby/3.0.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1008:in `route!'
/app/.bundle/gems/ruby/3.0.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1129:in `block in dispatch!'
/app/.bundle/gems/ruby/3.0.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `block in invoke'
/app/.bundle/gems/ruby/3.0.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `catch'
/app/.bundle/gems/ruby/3.0.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `invoke'
/app/.bundle/gems/ruby/3.0.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1124:in `dispatch!'
/app/.bundle/gems/ruby/3.0.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:939:in `block in call!'
/app/.bundle/gems/ruby/3.0.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `block in invoke'
/app/.bundle/gems/ruby/3.0.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `catch'
/app/.bundle/gems/ruby/3.0.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `invoke'
/app/.bundle/gems/ruby/3.0.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:939:in `call!'
/app/.bundle/gems/ruby/3.0.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:929:in `call'
/app/.bundle/gems/ruby/3.0.0/bundler/gems/prometheus-client-fc179858e6e0/lib/prometheus/middleware/exporter.rb:32:in `call'
/app/lib/middleware.rb:10:in `call'
/app/.bundle/gems/ruby/3.0.0/gems/secure_headers-6.3.3/lib/secure_headers/middleware.rb:11:in `call'
/app/.bundle/gems/ruby/3.0.0/gems/rack-ssl-enforcer-0.2.9/lib/rack/ssl-enforcer.rb:52:in `call'
/app/.bundle/gems/ruby/3.0.0/gems/rack-2.2.3/lib/rack/deflater.rb:44:in `call'
/app/.bundle/gems/ruby/3.0.0/gems/rack-protection-2.1.0/lib/rack/protection/xss_header.rb:18:in `call'
/app/.bundle/gems/ruby/3.0.0/gems/rack-protection-2.1.0/lib/rack/protection/path_traversal.rb:16:in `call'
/app/.bundle/gems/ruby/3.0.0/gems/rack-protection-2.1.0/lib/rack/protection/json_csrf.rb:26:in `call'
/app/.bundle/gems/ruby/3.0.0/gems/rack-protection-2.1.0/lib/rack/protection/base.rb:50:in `call'
/app/.bundle/gems/ruby/3.0.0/gems/rack-protection-2.1.0/lib/rack/protection/base.rb:50:in `call'
/app/.bundle/gems/ruby/3.0.0/gems/rack-2.2.3/lib/rack/logger.rb:17:in `call'
/app/.bundle/gems/ruby/3.0.0/gems/rack-2.2.3/lib/rack/common_logger.rb:38:in `call'
/app/.bundle/gems/ruby/3.0.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:253:in `call'
/app/.bundle/gems/ruby/3.0.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:246:in `call'
/app/.bundle/gems/ruby/3.0.0/gems/rack-2.2.3/lib/rack/head.rb:12:in `call'
/app/.bundle/gems/ruby/3.0.0/gems/rack-2.2.3/lib/rack/method_override.rb:24:in `call'
/app/.bundle/gems/ruby/3.0.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:216:in `call'
/app/.bundle/gems/ruby/3.0.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1991:in `call'
/app/.bundle/gems/ruby/3.0.0/gems/puma-5.5.2/lib/puma/configuration.rb:249:in `call'
/app/.bundle/gems/ruby/3.0.0/gems/puma-5.5.2/lib/puma/request.rb:77:in `block in handle_request'
/app/.bundle/gems/ruby/3.0.0/gems/puma-5.5.2/lib/puma/thread_pool.rb:340:in `with_force_shutdown'
/app/.bundle/gems/ruby/3.0.0/gems/puma-5.5.2/lib/puma/request.rb:76:in `handle_request'
/app/.bundle/gems/ruby/3.0.0/gems/puma-5.5.2/lib/puma/server.rb:447:in `process_client'
/app/.bundle/gems/ruby/3.0.0/gems/puma-5.5.2/lib/puma/thread_pool.rb:147:in `block in spawn_thread'
something broke. rssbox responds correctly with the URL of the feed, but the feed is "Something went wrong. Try again later."
Digging into this bit with a local checkout of the code it looks like Instagram's API might have changed. It seems the posts I'm getting back do not have the structure that RSS Box is expecting - RSS Box is looking for a top level "graphql" key in the JSON but what I'm getting for posts has this structure at the top:
{
"items": [...]
"num_results": 1,
"more_available": false,
"auto_load_more_enabled": false
}
I'm not familiar enough with the Instagram API to know if this is a known or a new response format. Can probably come up and dig a little deeper soon.
Hey, sorry everyone that I didn't take a closer look earlier. Looks like this problem only happened when you were using a sessionid (rssbox.herokoapp.com currently isn't).
I just pushed 7e0e92ac582d265f2356d5bb34fa1b422c32fe2f which I believe should fix the problem. A new docker image has also been pushed out.
Let me know if that works for ya. Happy valentine's day (there's 22 minutes left for me!).
yep. this seems to have worked. thank you <3 although, after the bulk update started, Instagram started returning 502 after the 20th feed :)
I've been running into a 429 errors pretty frequently recently. I'm following 10s of different accounts, but restarting my dyno (on Heroku) seems to be the culprit as the file cache will blown away, and RSS Box will then hit Instagram for all my accounts when I open my feed reader.
I'd be interested in experimenting with adding a throttling mechanic to prevent the requests happing in quick succession. Personally, I don't mind if things are a little out of date, so would be happy to throttle to one request per minute or potentially something even slower. Maybe this would end up being an "opt in" feature. I had a couple of questions about caching however to get my up to speed (most likely for @stefansundin):
- I was surprised that Redis wasn't being used for caching (I'd assumed that's what it was there for, but it looks like caching is file based). What is Redis used for?
- Looking at this line, it seems the intention is for Instagram feeds to be cached for a week (
7*24*60*60). Is that correct? I feel like I've seen updates come in faster than that, so that surprised me.