medium-crawler icon indicating copy to clipboard operation
medium-crawler copied to clipboard

Published 20 hours ago • verified publisher icon NISH1001

[Crawler Issue] Now the crawler doesn't fetch anything

Open NISH1001 opened this issue 5 years ago • 4 comments

As of writing this, it seems that there have been changes in div and other HTML elements in "medium dot com". This renders the crawler worthless for now. I will fix this when I have some leisure time to invest in this project.

:)

NISH1001 avatar Jul 25 '19 09:07 NISH1001

Any updates? :(

VladislavSmolyanoy avatar Feb 21 '23 19:02 VladislavSmolyanoy

@VladislavSmolyanoy Hi. I apologize but I have stopped working on this project. Maybe you can fork and try changing the crawler? I am sure there are better alternatives than this. Something like scrapy might do? :)

NISH1001 avatar Feb 22 '23 14:02 NISH1001

I was just trying to save some time and find a library that does that... I guess I'll have to build something myself then 😊

VladislavSmolyanoy avatar Feb 22 '23 14:02 VladislavSmolyanoy

I think you can start with scrapy. The only caveat is how you can bypass the paywall on medium. Medium today has become a mess of UI elements.

NISH1001 avatar Feb 22 '23 14:02 NISH1001