pl-predictions-using-fifa
pl-predictions-using-fifa copied to clipboard
BradleyGrantham
fifastats spider not working properly
Dear Bradley, I have an issue that the fifastats spider is not working properly. I checked and the problem is in the parse_player method, where I get IndexError index out of range when gathering name parameter. I checked the script and until that point it works properly it seems that the response html structure have changed or something like that happened since the name is empty. Could you validate that this is the problem, please? Impressed by your work, Greg
Hi Greg. Sorry about this. You are correct, the response has changed since when I orginally made it! Not sure when I will be able to look at it unfortunately! Bradley
The page markup must have changed completely. Scraper code makes no sense. What is worse, the new markup does not have good class hooks for Scapy to use. I'm looking into if its easily fixable...
Hi @randomm and @BradleyGrantham I updated the scraper to be able to scrape the new website. Happy to share the new code, also parsing the details of the players' scores (not only the overall one).
@javpascal That sounds great, is the code on GitHub? You're also more than welcome to PR it into here if you want, completely up to you though
That sounds great @javpascal, well done! ... why don't you do a pull request here?
Hi @BradleyGrantham see attached the updated crawler - it probably will be easier for you to reuse this one. Please note that the outputs of the "parse_player" function now include multiple columns, instead of the overall score only. You will need to select that column specifically.
Dear @javpascal! Thank you for sharing. However, I couldn't manage to get it working. It says that there is a list index out of range error during scraping when dealing with the nationality_name part. Am I the only one having this issue?
I ran into the same issues and so I updated the scraper. I made a pull request if someone is still interested. Nevertheless it is still a lot of work to gather all the data, to have it in the right folders and to get the rest of the code running.
