cisticola
cisticola copied to clipboard
Performance optimizations and bug fixes
- Modified langdetect detect method to decrease run-time
- Fixed indentation error in
transform_info - Prototyped removal of offset in
transform_all_untransformed- This change needs modifications: it fails for the first batch, since the
batchis already computed, so that only oneScraperResult.dateis greater than or equal tomax(batch, key=lambda v: v.date).date, so only one post is transformed.
- This change needs modifications: it fails for the first batch, since the
- Originally modified
TelegramTelethonTransformerto have aself.clientattribute, but this caused the transformer tests to fail, since having aTelegramTelethonScraperalready initialized while initializing aTelegramTelethonTransformercauses the Telethon session database to be locked. This could be addressed by deleting thecontrollerobject intests/transformer/telegram_telethon.py