IndexError: list index out of range (comments)

[marioacarvalho]

table = soup.findChildren('table')[3] # the table holding all comments IndexError: list index out of range

[marioacarvalho]

I would help you with pull requests but my knowledge in Python it's not so good..

[karan]

Can you post the traceback, and the story you were trying to scrape? Or the page?

[marioacarvalho]

id 7028714

2014-01-09T13:06:20.697732+00:00 app[web.1]: 2014-01-09 13:06:20 [7] [ERROR] Error handling request 2014-01-09T13:06:20.697732+00:00 app[web.1]: Traceback (most recent call last): 2014-01-09T13:06:20.697732+00:00 app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/gunicorn/workers/sync.py", line 131, in handle_request 2014-01-09T13:06:20.697732+00:00 app[web.1]: respiter = self.wsgi(environ, resp.start_response) 2014-01-09T13:06:20.697732+00:00 app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/flask/app.py", line 1836, in call 2014-01-09T13:06:20.697732+00:00 app[web.1]: return self.wsgi_app(environ, start_response) 2014-01-09T13:06:20.697732+00:00 app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/flask/app.py", line 1820, in wsgi_app 2014-01-09T13:06:20.697732+00:00 app[web.1]: response = self.make_response(self.handle_exception(e)) 2014-01-09T13:06:20.697732+00:00 app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/flask/app.py", line 1403, in handle_exception 2014-01-09T13:06:20.697732+00:00 app[web.1]: reraise(exc_type, exc_value, tb) 2014-01-09T13:06:20.698726+00:00 app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app 2014-01-09T13:06:20.698726+00:00 app[web.1]: response = self.full_dispatch_request() 2014-01-09T13:06:20.698726+00:00 app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request 2014-01-09T13:06:20.698726+00:00 app[web.1]: rv = self.handle_user_exception(e) 2014-01-09T13:06:20.698726+00:00 app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception 2014-01-09T13:06:20.698726+00:00 app[web.1]: reraise(exc_type, exc_value, tb) 2014-01-09T13:06:20.698726+00:00 app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request 2014-01-09T13:06:20.698726+00:00 app[web.1]: rv = self.dispatch_request() 2014-01-09T13:06:20.698726+00:00 app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/flask_debugtoolbar/init.py", line 125, in dispatch_request 2014-01-09T13:06:20.698726+00:00 app[web.1]: return view_func(**req.view_args) 2014-01-09T13:06:20.698892+00:00 app[web.1]: File "/app/hello.py", line 67, in updateHNFront 2014-01-09T13:06:20.698892+00:00 app[web.1]: front_posts = parseNewsFront() 2014-01-09T13:06:20.698892+00:00 app[web.1]: File "/app/hello.py", line 188, in parseNewsFront 2014-01-09T13:06:20.698892+00:00 app[web.1]: comments = story.get_comments() 2014-01-09T13:06:20.698892+00:00 app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/hn/hn.py", line 253, in get_comments 2014-01-09T13:06:20.698892+00:00 app[web.1]: return self._build_comments(soup) 2014-01-09T13:06:20.698892+00:00 app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/hn/hn.py", line 219, in _build_comments 2014-01-09T13:06:20.698892+00:00 app[web.1]: level = int(row.findChildren('td')[1].find('img').get('width')) // 40 2014-01-09T13:06:20.698892+00:00 app[web.1]: AttributeError: 'NoneType' object has no attribute 'get'

[karan]

It's because of the poll. I'll try to add poll support some time later.

[gauravaror]

<Story: ID=-1> Traceback (most recent call last): File "./my_test_bot.py", line 26, in comments = story.get_comments() File "/home/gaurav/Work/HackerNews/HackerNewsAPI/hn/hn.py", line 361, in get_comments return self._build_comments(soup) File "/home/gaurav/Work/HackerNews/HackerNewsAPI/hn/hn.py", line 229, in _build_comments table = soup.findChildren('table')[3] IndexError: list index out of range

Seems like the storyid is retured as -1 sometimes and that results in list index out of range. This maybe the additional case