Retrieve loaded data as pandas.DataFrame

[leferrad]

I think this library could be a great alternative to pyarrow and pyspark to easily read and write ORC files without requiring a big library to just achieve that (not to mention the current issues that pyarrow is having, and the overhead of loading a SparkSession just to read/write data). Therefore, in order to make it usable in most of data processing application, an easy connection with Pandas (the most popular library for local processing of tabular data) would be convenient. I'm new using this library, but I was evaluating two options:

1. As a method, to indicate that data should be retrieved as a pandas.DataFrame (that requires adding pandas as a dependency, which may not be desired)
2. Just an example on the documentation, to let interested users understand how a ORC file could be loaded as a pandas.DataFrame.

So far, I've solved that through this snippet (from what I could understand of the library):

import pandas as pd
import pyorc 
 
path_to_data =  "path/to/data.orc"

with open(path_to_data, "rb") as f:
    reader = pyorc.Reader(f) 
    columns = reader.schema.fields
    # sort by column id to ensure correct order (since "fields" is a dict, order may not be correct)
    columns = [y for x, y in sorted([(reader.schema.find_column_id(c), c) for c in columns])] 
    df = pd.DataFrame(reader, columns=columns)

[noirello]

Honestly, I'm not that familiar with pandas. I have to look into more, but the simplest solution to add pandas as an extra to the module, and inherit a new Reader/Writer from the existing ones with methods that can read and write pandas dataframes. There's probably a need for special converter functions for pandas' special types as well.

As you've already noticed, my goal with this library is to be a simple ORC reader and writer with the smallest overhead as possible. A few smaller tasks are on my todo list now, but I'm not against the idea. The best would be, if someone with a better knowledge of pandas could contribute. 😉

[leferrad]

Therefore, what do you suggest to solve this issue? These are the options I guess:

1. PandasReader & PandasWriter (I don't like this one)
2. "as_pandas" as method or argument on reader (but this adds pandas as dependency)
3. Just an example of how to get a Pandas DF from loaded data.
4. Nothing to do (just add a comment to let users understand the scope of pyorc)

Let me know what is your concern on this and then I could make a PR related to that.

[noirello]

You're probably right, separate classes for reading and writing are overkill. Maybe two simple functions defined in a submodule could be sufficient enough.

I don't want to add pandas as a default requirement to the module.

[leferrad]

OK so you prefer to add functions/methods for that, but not to require pandas to use pyorc. I can add this behavior without adding Panda as a requirement, and raise an error if Pandas is not installed, but that will be a very bad practice. Therefore, the best option will be to just add an example with Pandas for start, and if the example is not enough you could consider expanding the scope of pyorc to be more integrated with Pandas (which is my suggestion, since it's a pretty common requirement in most of data processing libraries). Let me know your thoughts and I can make a PR from that (either adding an example or adding methods with Pandas)

[noirello]

I think an example would be great as a start. A PR about that would be much appreciated. Thank you.

[fehtemam]

Can someone please include a short example of how to use converters in the Reader? I tried really hard to figure this out but I couldn't. I'm reading a file like this where orc_bytes is of class bytes: orc = pyorc.Reader(fileo=io.BytesIO(orc_bytes)) This works fine and I can convert the resulting Reader object to a pandas dataframe. Now I am trying to add a converter to Reader to convert decimal to float upon reading. I know it needs a dictionary with keys being TypeKind but I can't figure out how to pass the dictionary values. So I am stuck at: orc = pyorc.Reader(fileo=io.BytesIO(orc_bytes), converters={pyorc.TypeKind.DECIMAL: ???}) I found an example in test_reader.py file (https://github.com/noirello/pyorc/blob/master/tests/test_reader.py#L325) that uses ORCConverter to define a class and a from_orc method for TypeKind.TIMESTAMP but I have no idea how this should be defined to convert decimal to float. Any help please?

[fehtemam]

This is what I have so far but it is not doing any conversion:

import numpy as np
import pyorc
from pyorc.converters import ORCConverter

class TypeConverter(ORCConverter):
    @staticmethod
    def from_orc(decimal_input):
        return np.array(decimal_input, dtype=float)

orc = pyorc.Reader(fileo=io.BytesIO(orc_bytes), converters={pyorc.TypeKind.DECIMAL: TypeConverter})

Any suggestions?!

[noirello]

I updated the docs about ORCConverter

Your converter above should return a numpy array with a float in it, for every item in a decimal ORC column.

[fehtemam]

@noirello Thanks a lot! Highly appreciated.