asammdf
asammdf copied to clipboard
asammdf - pyspark integration
Bring support for pyspark integration with asammdf.
How would it work?
I would give a list of paths of MDF files I want to execute on all the MDF files my python logic in parallel using Spark
@Spratiher9 spark to deal with mdf files ,can you show me your code?
@shangrilaer For now What I do is have a spark data frame with just one column which contains the paths of the mdf files and then basically do
def f(row):
'''
some logic involving
python code asammdf
which basically does
parallel processing on
MDF files parallelly on
executors
'''
df.rdd.map(lambda row: f(row))