"IndexError: list index out of range" in _url_dedup_sql on finish.

[ilyaevseev]

On console:

# /home/rkn/blacklist-rkn-tool/bl-rkn.py --dump
Traceback (most recent call last):
  File "/home/rkn/blacklist-rkn-tool/bl-rkn.py", line 653, in <module>
    main()
  File "/home/rkn/blacklist-rkn-tool/bl-rkn.py", line 649, in main
    BlrknCLI()
  File "/home/rkn/blacklist-rkn-tool/bl-rkn.py", line 581, in __init__
    self._get_dump()
  File "/home/rkn/blacklist-rkn-tool/bl-rkn.py", line 609, in _get_dump
    message = self.report.statistics_show()
  File "/home/rkn/blacklist-rkn-tool/bl-rkn.py", line 104, in statistics_show
    url_add_sql = self._url_dedup_sql(diff, 'ignore', 1)
  File "/home/rkn/blacklist-rkn-tool/bl-rkn.py", line 437, in _url_dedup_sql
    url_diff_sql = URL.select(fn.Distinct(URL.url)).where(URL.add == self.idx_list[diff])
IndexError: list index out of range

In logs:

2018-04-04 23:51:48,970  Starting script.
2018-04-04 23:51:49,160  Check database: PostgreSQL Ok
2018-04-04 23:51:51,078  Current versions: webservice: 3, dump: 2.2, doc: 4
2018-04-04 23:51:51,079  New webservice: 3.1
2018-04-04 23:51:51,083  New dumpFormatVersion: 2.3
2018-04-04 23:51:51,086  New docVersion: 4.8
2018-04-04 23:51:53,789  Send email from ... to ...
2018-04-04 23:51:53,790  Current webservice:3
New webservice: 3.1

Current dumpFormatVersion: 2.2
New dumpFormatVersion: 2.3

Current docVersion: 4
New docVersion: 4.8


2018-04-04 23:51:53,790  Check if dump.xml has updates since last sync.
2018-04-04 23:51:53,794  Current date: lastDumpDate: 2012-01-01 00:00:00, lastDumpDateUrgently: 2012-01-01 00:00:00
2018-04-04 23:51:53,794  Last date: lastDumpDate: 2018-04-04 22:56:00, lastDumpDateUrgently: 2018-04-04 22:56:00
2018-04-04 23:51:53,794  New dump is available.
2018-04-04 23:51:53,798  Sending request.
2018-04-04 23:51:55,735  Checking request status.
2018-04-04 23:51:55,735  Got code 245874f427a272a4deab875eba47fe24
2018-04-04 23:51:55,741  Save code in History
2018-04-04 23:51:55,744  Waiting for a 90 sec.
2018-04-04 23:53:25,823  Trying to get result...
2018-04-04 23:53:28,747  Got a dump ver. 2.3 for the ...
2018-04-04 23:53:28,811  Downloaded dump 6826514 bytes, MD5 hashsum: 53ab1e693f36707235bb42642e8f3941
2018-04-04 23:53:28,811  Unpacking.
2018-04-04 23:53:29,314  dump.xml already exists.
2018-04-04 23:53:30,345  Got updateTime: 1522893360.
2018-04-04 23:53:30,347  Got updateTimeUrgently: 1522893360.
2018-04-04 23:53:30,908  New Item, IP, Domain, URL id: 786432.
2018-04-04 23:53:31,020  New Item, IP, Domain, URL id: 786433.

...

2018-04-05 01:59:41,953  New Item, IP, Domain, URL id: 786430.
2018-04-05 01:59:42,059  New Item, IP, Domain, URL id: 786431.
2018-04-05 02:02:44,339  Dump cleaner run
2018-04-05 02:02:44,341  Item deleted: 0
2018-04-05 02:02:44,342  IP deleted: 0
2018-04-05 02:02:44,343  Domain deleted: 0
2018-04-05 02:02:44,345  URL deleted: 0
2018-04-05 02:02:44,347  History deleted: 1

[Prototype-X]

Приветствую. У меня сейчас нет действующего сертификата, что бы посмотреть в чем проблема. Из логов видно что проблема началась после обновления версии дампа?

[ilyaevseev]

Проблема началась при первом запуске и повторяется при всех последующих.

[Prototype-X]

Какая версия Postgres psql -V? Какая версия peewee pip3 show peewee? Какая версия python python3 -V? Устанавливали релиз версию утилиты или последнюю git версию?

[ilyaevseev]

# psql -V
psql (PostgreSQL) 9.6.7

# pip3 show peewee
Name: peewee
Version: 3.2.2
Summary: a little orm
Home-page: http://github.com/coleifer/peewee/
Author: Charles Leifer
Author-email: [email protected]
License: UNKNOWN
Location: /usr/local/lib/python3.5/dist-packages
Requires: 

# python3 -V
Python 3.5.3

Версия blacklist-rkn-tool = 2.1.1