mongo-connector
mongo-connector copied to clipboard
StopIteration Runtime Exception with Python 3.8
`2021-04-22 07:36:48,231 [CRITICAL] mongo_connector.oplog_manager:713 - Exception during collection dump
Traceback (most recent call last):
File "/home/al/dev/zapper/projects/etl2/lib/python3.8/site-packages/mongo_connector/doc_managers/solr_doc_manager.py", line 292, in
The above exception was the direct cause of the following exception:
Traceback (most recent call last): File "/home/al/dev/zapper/projects/etl2/lib/python3.8/site-packages/mongo_connector/oplog_manager.py", line 668, in do_dump upsert_all(dm) File "/home/al/dev/zapper/projects/etl2/lib/python3.8/site-packages/mongo_connector/oplog_manager.py", line 651, in upsert_all dm.bulk_upsert(docs_to_dump(from_coll), mapped_ns, long_ts) File "/home/al/dev/zapper/projects/etl2/lib/python3.8/site-packages/mongo_connector/util.py", line 33, in wrapped return f(*args, **kwargs) File "/home/al/dev/zapper/projects/etl2/lib/python3.8/site-packages/mongo_connector/doc_managers/solr_doc_manager.py", line 292, in bulk_upsert batch = list(next(cleaned) for i in range(self.chunk_size)) RuntimeError: generator raised StopIteration 2021-04-22 07:36:48,232 [ERROR] mongo_connector.oplog_manager:723 - OplogThread: Failed during dump collection cannot recover!`
code fix is in solr_doc_manager.py, replace 2x instances of:
batch = list(next(cleaned) for i in range(self.chunk_size))
with
batch = [] for i in range(self.chunk_size): for x in cleaned: batch.append(x)