learning-spark
learning-spark copied to clipboard
Example 4-12 - PerKeyAvg for Python Incorrect
In the example, the map
method shows to take a lambda
with two parameters (key
and xy
), but it appears as though the python version of spark only has a map
method that expects a lambda with just a single parameter.
So instead of the following
r = sumCount.map(lambda key, xy: (key, xy[0]/xy[1])).collectAsMap()
We should use
r = sumCount.map( lambda kvp: ( kvp[0], kvp[1][0] / kvp[1][1] ) ).collectAsMap()
Note: It appears the github source foregoes the mapping step and returns a list of {key: (sum, count)} instead of {key: avg}.