instascrape
instascrape copied to clipboard
chris-greening
JSONDecodeError within scrape() method
Hi everyone!
Got an JSONDecodeError error after running the following code. I suspect it has to do with the source recognition as soup rather than string
from instascrape import *
'# Instantiate the scraper objects google = Profile('https://www.instagram.com/google/') google_post = Post('https://www.instagram.com/p/CG0UU3ylXnv/') google_hashtag = Hashtag('https://www.instagram.com/explore/tags/google/')
'# Scrape their respective data google.scrape() google_post.scrape() google_hashtag.scrape()
print(google.followers) print(google_post['hashtags']) print(google_hashtag.amount_of_posts)
JSONDecodeError
JSONDecodeError Traceback (most recent call last)
~\anaconda3\lib\site-packages\instascrape\core_static_scraper.py in scrape(self, mapping, keys, exclude, headers, inplace, session, webdriver) 142 scraped_dict = self.source.to_dict() 143 else: --> 144 return_data = self._get_json_from_source(self.source, headers=headers, session=session) 145 flat_json_dict = flatten_dict(return_data["json_dict"]) 146
~\anaconda3\lib\site-packages\instascrape\core_static_scraper.py in _get_json_from_source(self, source, headers, session) 258 if initial_type: 259 soup = self.source --> 260 json_dict_arr = json_from_soup(soup) 261 if len(json_dict_arr) == 1: 262 json_dict = json_dict_arr[0]
~\anaconda3\lib\site-packages\instascrape\scrapers\scrape_tools.py in json_from_soup(source, as_dict, flatten) 92 93 if as_dict: ---> 94 json_data = [json.loads(json_str) for json_str in json_data] 95 if flatten: 96 json_data = [flatten_dict(json_dict) for json_dict in json_data]
~\anaconda3\lib\site-packages\instascrape\scrapers\scrape_tools.py in
~\anaconda3\lib\json_init_.py in loads(s, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw) 355 parse_int is None and parse_float is None and 356 parse_constant is None and object_pairs_hook is None and not kw): --> 357 return _default_decoder.decode(s) 358 if cls is None: 359 cls = JSONDecoder
~\anaconda3\lib\json\decoder.py in decode(self, s, _w) 335 336 """ --> 337 obj, end = self.raw_decode(s, idx=_w(s, 0).end()) 338 end = _w(s, end).end() 339 if end != len(s):
~\anaconda3\lib\json\decoder.py in raw_decode(self, s, idx) 351 """ 352 try: --> 353 obj, end = self.scan_once(s, idx) 354 except StopIteration as err: 355 raise JSONDecodeError("Expecting value", s, err.value) from None
JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)
To Reproduce Steps to reproduce the behavior:
- Run the example provided here https://github.com/chris-greening/instascrape
- See error described above
Screenshots If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):
- OS: Windows 10
i tried going through this with the debugger.
I don't know the original intent of this code, or how it worked before https://github.com/chris-greening/instascrape/blob/master/instascrape/core/json_algos.py#L75
but basically the json strings returned by this filter are not all pure json data, but also contain javascript code. Checking for any script tag with 'config' in it is not the correct way to find json script tags, at least anymore. I tried lookin at the source html and couldn't figure out which json data is actually needed. The media html is using a bunch of async loading of json data embedded in javascript code, as well, that will be difficult to parse correctly, probably designed to make scraping the website very difficult without using the Facebook developer API. I couldn't even figure out how to manually download a single video from a reel page (i.e. using chrome developer tools / inspector) either. I suppose long gone are the days of just searching for a ".mp4" link in the source and downloading it, with all the obfuscation and anti-scraping / anti-downloading going on...
There hasnt been any commit on this package in over a year, probably have to fix it our selves / fork it if the developers are no longer active.
edit: after a big more digging around, finally found the mp4 sources for reels in the network profiler with some url editing... The download speed is super slow, and the audio / video are split into separate files, but this is good enough for me, at least for now.
Thank you for your quick response.
Do you know how efficient scrapping with Facebook dev API is? I'd appreciate any resource that you can provide.
On Thu, Jul 28, 2022 at 9:43 AM Stephen Hopkins @.***> wrote:
i tried going through this with the debugger.
I don't know the original intent of this code, or how it worked before https://github.com/chris-greening/instascrape/blob/master/instascrape/core/json_algos.py#L75
but basically the json strings returned by this filter are not all pure json data, but also contain javascript code. Checking for any script tag with 'config' in it is not the correct way to find json script tags, at least anymore. I tried lookin at the source html and couldn't figure out which json data is actually needed. The media html is using a bunch of async loading of json data embedded in javascript code, as well, that will be difficult to parse correctly, probably designed to make scraping the website very difficult without using the Facebook developer API.
There hasnt been any commit on this package in over a year, probably have to fix it our selves / fork it if the developers are no longer active.
— Reply to this email directly, view it on GitHub https://github.com/chris-greening/instascrape/issues/164#issuecomment-1197783711, or unsubscribe https://github.com/notifications/unsubscribe-auth/A2HQJCULX3YIJRJDWOYH4UTVWI2ZPANCNFSM54VZVNIQ . You are receiving this because you authored the thread.Message ID: @.***>
I don't know too much about the API, except it is only usable by registered and most likely required to be approved, facebook integrated applications.
