data_analysis

data_analysis copied to clipboard

boss_spider中的cookies失效问题

3

1.大致在shell中按里面的代码运行了爬虫，爬取多页时发现，几页后返回的信息就不对了； 2.查看浏览器network项中的信息，发现request cookies每次刷新都会变化，cookies的expires/max-age项为N/A。尝试在爬虫中更新cookie，但是response.cookies里面没有新的cookies； 3.某一次刷新得到的cookies如下（与config.py中的cookies不同）： lastCity, 101280600 __c, 1570767132 __g, - __l, l=%2Fwww.zhipin.com%2F&r=&friend_source=0&friend_source=0 __zp_stoken__, 1f9cwxB9fG2zYF9YsVAIU%2F2z12UYeEyWl5XZdq9jBSY4%2FL7WJc63GzWwGHp0PtQv1EUjW1CzPijL6y11S2RHdM7xKQ%3D%3D __a, 89073411.1570767132..1570767132.130.1.130.130 Hm_lvt_194df3105ad7148dcf2b98a91b5e727a, 1570767132,1572024861 Hm_lpvt_194df3105ad7148dcf2b98a91b5e727a, 1572025776

Paradiseeee

table和tr找不到

5

def get_team_data(): qiudui_url = 'https://www.dongqiudi.com/data?competition=8' qiudui_res = requests.get(qiudui_url, headers=header, cookies=session).text content = BeautifulSoup(qiudui_res, 'html.parser') team_content = content.find('**table**').find_all('**tr**') team_list = list(map(deal_element_list, team_content[2:])) save_to_csv(team_list) print('get player data now...') for i in team_list:...

messi-fcb-10