Avoiding Overflow Error when Building Index on Windows

[greenguy33]

Thank you for this great library, I have found it very useful!

If I run the build index command as specified in the README on Windows using the library binaries, I encounter the following error: OverflowError: Python int too large to convert to C long Here are steps I took to run the program on Windows:

1. Clone the repository locally
2. In cli.py, add this code so it can be run as a standalone Python program:

if __name__ == "__main__":
    main()

3. In cli.py, change the import to use the local version of processor.py:

from wikimapper import download_wikidumps, WikiMapper
from processor import create_index

4. In processor.py, comment out line 117:

#csv.field_size_limit(sys.maxsize)

Then you can run the code locally in this way:

cli.py create enwiki-latest --dumpdir data --target data/index_enwiki-latest.db

Hope it helps someone!

[jcklie]

Thanks for the issue report! I fixed it and added tests for Windows. I also released 0.1.6 which contains the fix. Please close the bug if that resolves the issue.