Empty loop in get_remote_tweets leads to error (variable referenced before assignment)

[kdave]

> twtxt timeline
Traceback (most recent call last):
  File "/usr/bin/twtxt", line 9, in <module>
    load_entry_point('twtxt==1.2.1', 'console_scripts', 'twtxt')()
  File "/usr/lib/python3.4/site-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3.4/site-packages/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3.4/site-packages/click/core.py", line 1060, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3.4/site-packages/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3.4/site-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python3.4/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/lib/python3.4/site-packages/twtxt/cli.py", line 126, in timeline
    tweets = get_remote_tweets(sources, limit, timeout, cache)
  File "/usr/lib/python3.4/site-packages/twtxt/twhttp.py", line 123, in get_remote_tweets
    return tweets
UnboundLocalError: local variable 'tweets' referenced before assignment

This started after I added a few new follow urls, I don't know which one does that, will try to find out.

[kdave]

Running directly from git:

2016-03-07 14:36:39,680 asyncio      ERROR    Unclosed response
client_response: <ClientResponse(https://twtxt.xvm.space/) [None None]>
None

[kdave]

https://twtxt.xvm.space/ appears in the failed request, but removing it from the config still produces the same error.

[kdave]

I've iterated 'twtxt view' over all my follows and it points to ... @mdom ! (https://mdom.github.io/twtxt.txt)

[mdom]

I added a comment line yesterday that seemed to be the culprit. It worked for @kdave after i removed the line... :)

[buckket]

@mdom Do you still know which line caused the error? Trying to reproduce this so I can catch the exception which leads to this.

[mdom]

This was my first experiment with comments: A hash sign at the beginning of a line. I think this line crashed twtxt.

[buckket]

I’m not able to reproduce this with 1.3.0dev.

[kdave]

I can see the same crash with current master (5cb114f6c95a9dce049), try on this url: https://kdave.github.io/twtxt-test.txt

[buckket]

Very strange, I get this: twtxt.parser DEBUG https://kdave.github.io/twtxt-test.txt - Unknown string format

But no exception, twtxt continues processing the other sources and tweets, as intended.

Can you provide the --verbose output?

[kdave]

top commit 2b31ad3cc496f63a257e2eb944de7dc6f4b8f7ad

$ twtxt -v view https://kdave.github.io/twtxt-test.txt
2016-04-04 10:27:04,956 twtxt.cli    DEBUG    Not following https://kdave.github.io/twtxt-test.txt, trying as URL
2016-04-04 10:27:04,958 asyncio      DEBUG    Using selector: EpollSelector
2016-04-04 10:27:04,959 asyncio      DEBUG    poll 0.000 took 0.000 seconds
2016-04-04 10:27:04,959 asyncio      DEBUG    poll 0.000 took 0.000 seconds
2016-04-04 10:27:04,962 asyncio      DEBUG    poll 0.000 took 0.000 seconds
2016-04-04 10:27:04,983 asyncio      DEBUG    poll 4.998 took 0.019 seconds
2016-04-04 10:27:04,983 asyncio      DEBUG    poll 0.000 took 0.000 seconds
2016-04-04 10:27:04,984 asyncio      DEBUG    poll 4.979 took 0.000 seconds
2016-04-04 10:27:04,984 asyncio      DEBUG    poll 0.000 took 0.000 seconds
2016-04-04 10:27:04,984 asyncio      DEBUG    poll 0.000 took 0.000 seconds
2016-04-04 10:27:05,008 asyncio      DEBUG    poll 4.978 took 0.024 seconds
2016-04-04 10:27:05,008 asyncio      DEBUG    poll 0.000 took 0.000 seconds
2016-04-04 10:27:05,035 asyncio      DEBUG    poll 4.953 took 0.026 seconds
2016-04-04 10:27:05,063 asyncio      DEBUG    poll 4.923 took 0.024 seconds
2016-04-04 10:27:05,064 asyncio      DEBUG    poll 0.000 took 0.000 seconds
2016-04-04 10:27:05,064 asyncio      DEBUG    poll 0.000 took 0.000 seconds
2016-04-04 10:27:05,064 asyncio      DEBUG    poll 0.000 took 0.000 seconds
2016-04-04 10:27:05,064 asyncio      DEBUG    poll 0.000 took 0.000 seconds
2016-04-04 10:27:05,065 asyncio      DEBUG    poll 0.000 took 0.000 seconds
2016-04-04 10:27:05,090 asyncio      DEBUG    poll took 0.025 seconds
2016-04-04 10:27:05,090 asyncio      DEBUG    poll 0.000 took 0.000 seconds
2016-04-04 10:27:05,095 asyncio      DEBUG    poll 0.000 took 0.000 seconds
2016-04-04 10:27:05,096 asyncio      DEBUG    poll 0.000 took 0.000 seconds
2016-04-04 10:27:05,096 asyncio      DEBUG    poll 0.000 took 0.000 seconds
Traceback (most recent call last):
  File "/usr/bin/twtxt", line 9, in <module>
    load_entry_point('twtxt==1.3.0.dev0', 'console_scripts', 'twtxt')()
  File "/usr/lib/python3.4/site-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3.4/site-packages/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3.4/site-packages/click/core.py", line 1060, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3.4/site-packages/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3.4/site-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python3.4/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/lib/python3.4/site-packages/twtxt/cli.py", line 195, in view
    ctx.forward(timeline)
  File "/usr/lib/python3.4/site-packages/click/core.py", line 552, in forward
    return self.invoke(cmd, **kwargs)
  File "/usr/lib/python3.4/site-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python3.4/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/lib/python3.4/site-packages/twtxt/cli.py", line 155, in timeline
    tweets = sort_and_truncate_tweets(tweets, sorting, limit)
UnboundLocalError: local variable 'tweets' referenced before assignment

[buckket]

Okay, using the same version with the exact same arguments I get:

$ twtxt -v view https://kdave.github.io/twtxt-test.txt
2016-04-06 15:28:35,265 twtxt.cli    DEBUG    Not following https://kdave.github.io/twtxt-test.txt, trying as URL
2016-04-06 15:28:35,290 asyncio      DEBUG    Using selector: EpollSelector
2016-04-06 15:28:35,576 twtxt.parser DEBUG    https://kdave.github.io/twtxt-test.txt - Unknown string format
2016-04-06 15:28:35,576 twtxt.twhttp DEBUG    https://kdave.github.io/twtxt-test.txt returned 200 and Last-Modified header - adding content to cache

➤ https://kdave.github.io/twtxt-test.txt (27 days ago):
fc6c8cdfe580 btrfs-progs: check: unify naming of long option values / David Sterba

I guess it has something to do with twtxt’s dependencies or even the Python version in use. I’ve tested it successfully with Python 3.4.4 and Python 3.5.1.

It worked with these dependencies installed:

aiohttp==0.21.5
chardet==2.3.0
click==6.6
coverage==4.0.3
humanize==0.5.1
pluggy==0.3.1
py==1.4.31
pytest==2.9.1
pytest-cov==2.2.1
python-dateutil==2.5.2
six==1.10.0
tox==2.3.1
virtualenv==15.0.1