Extending partial dependence for pyspark support

[pramitchoudhary]

The idea with the ticket is to explore the possibility of extending the support of PDP for MLLib models as well.

• As In-Memory Model
  • either using the existing implementation and extending the support spark DataFrames
  • native spark implementation for a more scalable and performant implementation
• As Deployed Model Depending on the complexity, there might be a need to split this task into sub tickets.
  • May we start here, to get the idea flowing and then moving into In-Memory support. Just a thought. Will have to think more about it.

Thoughts ? @benvandyke @aikramer2

[pramitchoudhary]

Also, not sure why 'Assignees' is not allowing to include you guys. Will figure it out.

[pramitchoudhary]

@benvandyke any luck with this one?

[benvandyke]

@pramitchoudhary Haven't had a chance to look into this yet, I've been doing more ETL and pipelines than modeling lately.