warp_prism icon indicating copy to clipboard operation
warp_prism copied to clipboard

ENH: Support python 2.

Open ssanderson opened this issue 8 years ago • 3 comments

Initial cut of support for python 2.

Notable deficiencies:

  • Removes keyword-only args from several API functions.

  • Changes BytesIO.getbuffer() to BytesIO.getvalue(). This is almost certainly a significant performance regression for large loads. At the very least, it should be made conditional by python version. It's an open question whether the extra copy incurred by calling getvalue() makes this even worth doing. If not, we'd probably have to vendor a py2-compat BytesIO-like object to support zero-copy reads.

ssanderson avatar May 01 '17 13:05 ssanderson

Not sure if this is actually worth doing, but, I was seeing unreasonable memory usage for fundamentals queries on Q over the weekend, and wanted to at least test to see if warp_prism improved that situation.

ssanderson avatar May 01 '17 13:05 ssanderson

In Python 2 cStringIO has a C API where you can read from the underlying buffer.

There is a capsule called cStringIO_CAPI which exports a structure with a read function. You can pass -1 to read the whole thing as a char*. This doesn't copy the data. This can't be merged as is because of the getvalue call.

Even with the getvalue call was this better for fundies?

llllllllll avatar May 01 '17 22:05 llllllllll

Even with the getvalue call was this better for fundies?

Don't know, I haven't actually used this yet other than getting the tests to pass in py2.

ssanderson avatar May 01 '17 22:05 ssanderson