MemoryError() using Windows version with large files

[mmascolino]

On 64bit Windows box with 64GBs of RAM I am working on a 1.8 GB csv file that has about 15 million rows of data. Even running a simple: q "SELECT COUNT(*) as cnt FROM mylargefile.csv" -d "," -H is resulting in a MemoryError()

Am I doing something wrong or is there a serious limitation here?

[harelba]

Hi sorry for the late reply, i'm on a long vacation...

There is a limitation on the data size that q can process in that case, since on windows it is currently packages with the 32 bit version of python.

I'm working on an experimental version which will be using Apache Spark behind the scenes and will allow to work around this kind of limitation (and provide much more speed in many use cases). Once it's out, this issue will no longer happen obviously.

I'm really sorry for the inconvenience and that I don't have a good solution for your needs.

[mmascolino]

Thanks for the response. I'd be happy to test something once you have something ready. Just let me know.