asammdf - pyspark integration

[Spratiher9]

Bring support for pyspark integration with asammdf.

[danielhrisca]

How would it work?

[Spratiher9]

I would give a list of paths of MDF files I want to execute on all the MDF files my python logic in parallel using Spark

[shangrilaer]

@Spratiher9 spark to deal with mdf files ,can you show me your code?

[Spratiher9]

@shangrilaer For now What I do is have a spark data frame with just one column which contains the paths of the mdf files and then basically do

def f(row):
  '''
  some logic involving 
  python code asammdf
  which basically does 
  parallel processing on
  MDF files parallelly on 
  executors
  '''

df.rdd.map(lambda row: f(row))