Otto Fajardo
Otto Fajardo
I am glad to hear that your problem is better now with the existing solution. Maybe you are right and there is still something to gain there, but it sounds...
also, just out of curiosity, you said initially the file took 25 hours to complete. How long is it taking now? what is the chunksize and how many cores are...
My thinking is that continuously giving chunks to multiprocessing will not help too much, because the idea of chunking is that you cannot handle the whole file in RAM and...
Implementing reading to a polars dataframe would be easy, as everything is in place. If you check the Readme it is described how to do it without passing through pandas....
thanks @mettekou that is a very interesting suggestion, I was not aware of this feature, it can indeed be a good solution to have an unified interface to deal with...
I am documenting some of my experiments here: First, I did a memory profiling using [memray](https://github.com/bloomberg/memray), writing a pandas dataframe to a SPSS file. First a very small dataframe of...
Thanks for the suggestion, I will look into it
hey, narwhals look really awesome! It looks really fit for purpose for the problem at hands here. I did a quick test, writing the large 1 Gb dataframe of integers...
Good news! Support for polars is ready on the branch test_narwhals! I would be very grateful if people could test it before releasing, both polars and pandas. You can either...
Thanks for testing! Would it be possible for you to share the file for me to debug? If not maybe you could create a dummy file to reproduce the error,...