bangalore-flat-and-flatmates-crawler icon indicating copy to clipboard operation
bangalore-flat-and-flatmates-crawler copied to clipboard

Repeating text content

Open aditeyabaral opened this issue 1 year ago • 0 comments

There seems to be repeating text content for some entries in the database. The entire body of text just repeats itself after ending. Can be fixed by checking by:

length = len(content)
if content[:length//2] == content[length//2:]:
        content = content[:length//2] 

This needs to be added to the processor

aditeyabaral avatar Oct 18 '22 20:10 aditeyabaral