pycraigslist
pycraigslist copied to clipboard
irahorecka
No results from query
The following Python code was returning accurate results for several months. In tests with results showing up properly on craigslist, the library is returning 0 results using the same query.
city = "newyork"
result = pycraigslist.gigs.cpg(site=city, posted_today=True, search_distance=100)
It appears Craigslist has added a redirect to the search resulting in adding something like this to the URL:
#search=1~list~0~0
Might this be affecting the library? I also noticed that when modifying searchNearby to 0 in the URL, the page returns results, but this is not able to be set using this library.
Because I needed the capability, I modified my solution to use pupeteer-cluster and it's working very well. I appreciate the energy that went into this library. Thank you and it served well for some time. šš¼
Hello - Iām sorry for the delay; for some reason I did not get this notification. I will take a look at rectifying this issue in a couple of weeks.
As an update, selenium for both Chrome and Firefox does not work; Craigslist is able to pick up the decoy browsers. @paradise-point How were you able to scrape with pupeteer? I tried this method and Craigslist still notices a phony browser / JavaScript enabled session.
Hello,
It definitely stopped working for me 2 day ago. I am only using the CraigslistForSale module. Is it something that can be fixed?
Thank you for your work.
@usctzen It is nice to hear from you again! I have not looked too closely at this recently due to other priorities, but it seems that Craigslist is really cracking down on web scrapers; even a browser automation tool such as Selenium is detected. I will peek at other similar repositories to see if they made any headway.
Edit: To add to this, it seems that North American sites and areas are the primary targets of Craigslist's anti-webscraping efforts. For example, sites tokyo and berlin seem to work fine. I tried suggestions in this Stackoverflow article but to no avail.
@irahorecka Thanks for the fast answer. I'll wait to see if you can fix it. I was using it for a friend and I am not sure it helped much for him. That being said, I'd love to make it work again.
Thanks for all your effort, your app is just too good to be forgotten.
@irahorecka don't know if you know about this https://blog.3d-logic.com/2022/10/02/craigslist-automation/