Packt Publishing website updated

[aojanzen]

Hi,

the website packtpub.com has been updated, the code in the notebook does not work any more. I found the field for the user name in the code inspector but was not able to find the right element qualifier to address it. The 2nd example with the pybites banner generator did of course not work either because Bob did not reveal his password. Can't you find (or create) a website that is not changed frequently and easily accessible for everyone?

Cheers, Andreas

[tazV2]

man! @aojanzen isnt kidding! I tried 100 days of python years ago but did not expect packt changed that much!

hello @aojanzen , you have to try xpath to find right elements. i spent/straggled sometime to make it work. Did you see in sign in page 'https://account.packtpub.com/login?returnUrl=referrer', there will be a notification popup? you have to click that. Here is the code for the packt part:

# usual stuff

driver = webdriver.Firefox()
driver.get('https://account.packtpub.com/login?returnUrl=referrer') # login page
time.sleep(5) # let it load

driver.find_element_by_xpath('//html/body/div[2]/div/div[3]/div[2]/button[1]').click() #get rid of the popup 

driver.find_element_by_xpath("//html/body/app-root/div/ng-component/div/div/ng-component/div/form/div[1]/input").send_keys('username')
driver.find_element_by_xpath('//html/body/app-root/div/ng-component/div/div/ng-component/div/form/div[2]/input').send_keys('password')
driver.find_element_by_xpath('//html/body/app-root/div/ng-component/div/div/ng-component/div/form/button').click() # click login 
 

@bbelderbos and @hobojoe1848, please update the course soon as possible

[bbelderbos]

Thanks @tazV2 @aojanzen, I will look at this today.

[bbelderbos]

Sigh, not only did the design change, you cannot download books anymore, and I needed a bunch of sleeps to actually get the content to load. This works for me (loading in my Packt credentials from my environment):

$ more script.py
import os
import time

from selenium import webdriver
from selenium.webdriver.common.keys import Keys

USERNAME = os.environ.get('PACKT_USER')
PASSWORD = os.environ.get('PACKT_PW')
PACKT_LOGIN = 'https://account.packtpub.com/login'

driver = webdriver.Chrome()
driver.get(PACKT_LOGIN)

# page is a bit slow
time.sleep(3)

driver.find_element_by_name('email').send_keys(USERNAME)
driver.find_element_by_name('password').send_keys(PASSWORD + Keys.RETURN)

# go to my books (make sure it's available)
time.sleep(3)
driver.find_element_by_link_text('My Owned Products').click()

# unfortunately you cannot download titles anymore :(
# again there is a slight delay
time.sleep(2)
elements = driver.find_elements_by_class_name("owned-product")
for e in elements:
    print(e.find_element_by_class_name('title').text)

Output (pagination so only first 10 results, no download links to go after):

$ python script.py
Mastering Python Networking - Second Edition
Docker Cookbook - Second Edition
Hands-On Dark Web Analysis
Modern JavaScript Web Development Cookbook
Hands-On Meta Learning with Python
Data Science Algorithms in a Week - Second Edition
Natural Language Processing with TensorFlow
Hands-On Machine Learning for Algorithmic Trading
Architects of Intelligence
Hands-On Transfer Learning with Python

So:

1. I think we add a disclaimer to the README or starting video that Packt's website changed so the code won't work anymore.

2. Maybe we can add another scraper to this repo, using something we host ourselves? In that case what would be cool?

cc @mikeckennedy @hobojoe1848 - what do you guys think?

[aojanzen]

Thank you, guys! I really appreciate the quick replies.