scalding
scalding copied to clipboard
Move ReferencedClassFinder to base
Ideally anything that does not increase the dependencies should be moved to base. By removing a function it looks like no one was using (and is just a trivial redirection to Config) we can do that here (although the tests do reference things in code).
cc @navinvishy @daniel-sudz I think we could use this code in the beam and spark backends in each stage of writes to try to create tokens for all the classes we can see. In kryo, this is registering classes so they have names. Note, I would sort the classes first so we assign the classes in a reproducible way that never depends on any iteration order of a hash set.
Codecov Report
Merging #1987 (fe647f8) into develop (6434348) will decrease coverage by
0.04%
. The diff coverage isn/a
.
@@ Coverage Diff @@
## develop #1987 +/- ##
=============================================
- Coverage 37.82% 37.78% -0.05%
- Complexity 1116 1335 +219
=============================================
Files 316 315 -1
Lines 21117 21061 -56
Branches 2924 2874 -50
=============================================
- Hits 7988 7957 -31
+ Misses 12141 12123 -18
+ Partials 988 981 -7
Impacted Files | Coverage Δ | |
---|---|---|
...a/com/twitter/scalding/ReferencedClassFinder.scala |
Continue to review full report at Codecov.
Legend - Click here to learn more
Δ = absolute <relative> (impact)
,ø = not affected
,? = missing data
Powered by Codecov. Last update 6434348...fe647f8. Read the comment docs.
In the beam backend, I see that we're getting a kryo instantiator here. We're using KryoHadoop
by default which gets Cascading serialization tokens and registers those classes with Kryo. Are you suggesting removing the dependency on KryoHadoop
?