learning-spark icon indicating copy to clipboard operation
learning-spark copied to clipboard

Example 4-12 - PerKeyAvg for Python Incorrect

Open funseiki opened this issue 8 years ago • 1 comments

In the example, the map method shows to take a lambda with two parameters (key and xy), but it appears as though the python version of spark only has a map method that expects a lambda with just a single parameter.

So instead of the following

r = sumCount.map(lambda key, xy: (key, xy[0]/xy[1])).collectAsMap()

We should use

 r = sumCount.map( lambda kvp: ( kvp[0], kvp[1][0] / kvp[1][1] ) ).collectAsMap()

funseiki avatar Aug 01 '16 21:08 funseiki

Note: It appears the github source foregoes the mapping step and returns a list of {key: (sum, count)} instead of {key: avg}.

funseiki avatar Aug 01 '16 21:08 funseiki