P. Oscar Boykin

Use scalafmt 0.5

1

We have tried this before. It now seems to work. This removes scalariform. scalafmt seems to be the new standard is actually being actively developed: https://olafurpg.github.io/scalafmt/ One change: this does...

Remove several implicits for TraversableOnce to Fields

16

addresses #1657 Note, `.isDefined` on fields is actually what you might expect: ```scala scala> import com.twitter.scalding.Dsl._ import com.twitter.scalding.Dsl._ scala> List().isDefined res0: Boolean = false scala> List(1).isDefined res1: Boolean = true...

``` [info] WritePartitionerTest: [info] - When we break at forks we have at most 2 + hashJoin steps *** FAILED *** [info] TestFailedException was thrown during property evaluation. [info] Message:...

Design doc on static evidence for grouping/serialization

1

@ttim has a design and even PR: #1857 to improve performance in scalding. The idea is to move towards requiring evidence that we can do binary sorting without deserializing needed...

partitioner sometimes violates the n + 1 rule

6

``` [info] - the total number of steps is not more than cascading *** FAILED *** [info] TestFailedException was thrown during property evaluation. [info] Message: 5 was not less than...

we have seen some issues with the scalding side-effect based counter API: see #1716 Scalding Operations use counters to log cache hit-rate effectiveness. We should just use the normal hadoop/cascading...

Add a spark backend

4

To test the modularity, a basic spark backend based on RDDs should be implemented. Should be easy to do to based on the cascading backend and the memory backend.

ArrayIndexOutOfBoundsException in MapTask

2

this error was observed running scalding 0.17.3. We do not yet know the cause. ``` at cascading.flow.hadoop.FlowMapper.run(FlowMapper.java:148) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:460) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:344) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)...

WritePartitioner step count law fails sometimes

1

We really want O(1) steps per partition, since we want to make sure cascading can plan it fast, but still we don't understand why the law is failing. It could...

We can take a series of writes in the Typed API, or we can use a FlowDef to create an Execution which has been broken into small pieces so cascading...