NimbusML icon indicating copy to clipboard operation
NimbusML copied to clipboard

Expose IDataView in python

Open ganik opened this issue 5 years ago • 0 comments

Currently when data is transformed as part of pipeline: pipeline.transform(data) the output is pandas dataframe.

It would be good to return a pointer to a IDataView object and realize it into a data later.

idataview = pipeline.transform(data)
...
while(idataview.move_next())
         data = idataview.read_next()

or read in batch:
while(idataview.move_next())
         data = idataview.read_next(batch_size)

this potentially would allow us to plugin our pipeline into other frameworks that operate on batches of data like pytorch.



ganik avatar Jun 17 '19 21:06 ganik