ScraperFC
ScraperFC copied to clipboard
oseymour
Transfermarkt Module Not Finding Tags
Module: Transfermarkt
Description of problem: Unable to retrieve information with transfermarkt module.
Code:
import ScraperFC as sfc
import traceback
scraper = sfc.Transfermarkt()
try:
# Scrape the table
player_links = scraper.get_players(year=2022, league="EPL")
except:
traceback.print_exc()
finally:
scraper.close()
Error message:
AttributeError Traceback (most recent call last) Cell In[7], line 4 1 import ScraperFC as sfc 2 import traceback ----> 4 scraper = sfc.Transfermarkt() 5 try: 6 # Scrape the table 7 player_links = scraper.get_player_links(year=2023, league="La Liga")
File ~/Desktop/github/scraperfc/ScraperFC/Transfermarkt.py:38, in Transfermarkt.init(self) 34 # Press the ACCEPT ALL button 35 soup = BeautifulSoup(self.driver.page_source, 'html.parser') 36 accept_all_button = self.driver.find_element( 37 By.XPATH, ---> 38 xpath_soup(soup.find('button', {'aria-label': 'ACCEPT ALL'})) 39 ) 40 self.driver.execute_script('arguments[0].click()', accept_all_button) 41 # Switch back to the main window
File ~/Desktop/github/scraperfc/ScraperFC/shared_functions.py:696, in xpath_soup(element) 667 """ Generate xpath from BeautifulSoup4 element. 668 669 I shamelessly stole this from https://gist.github.com/ergoithz/6cf043e3fdedd1b94fcf. (...) 693 "/doc/elm[2]" 694 """ 695 components = [] --> 696 child = element if element.name else element.parent 697 for parent in child.parents: # type: bs4.element.Tag 698 siblings = parent.find_all(child.name, recursive=False)
AttributeError: 'NoneType' object has no attribute 'name'
They must have changed their cookie accepting popup. I'm working on other projects right now but this is on the to-do list for when I have time for ScraperFC again!
@JoaoMCAlves haven't had a chance to look at this yet, sorry. Spent the last month prepping for ACL and meniscus surgery which happened yesterday. I'm planning on using my time laid up in bed to fix this.
Hello @oseymour , I'm sorry to hear that, I hope you have a speedy recovery. Congratulations on the excellent work you've been doing and thank you for the effort.
Yep, still working on all the other changes for the next release before it's pushed to PyPI. But if you really need the changes now they're on the unit tests branch.
No problem! I know that it always takes time. I just bought you a coffee 🙂