scalding icon indicating copy to clipboard operation
scalding copied to clipboard

Move parquet cascading schemes to subprojects

Open rubanm opened this issue 9 years ago • 11 comments

This moves all the parquet schemes to separate sub-projects than the parquet sources. This is mainly for easier upgrade to cascading3 and future versions perhaps. (As things stand now, we need to upgrade scalding-core in our cascading3 branch, before scalding-parquet for example.)

rubanm avatar Feb 11 '16 23:02 rubanm

:+1:

johnynek avatar Feb 12 '16 19:02 johnynek

This seems useful for the cascading 3 upgrade, and makes sense there. Do we want it in the 0.16 release cycle ? Or keep it closer to the big painful cascading one?

@rubanm would this be hard to take as an update at twitter? (just as an exemplar of issues elsewhere possibly).

ianoc avatar Feb 13 '16 00:02 ianoc

With the aliases and deps in place it should be handled mostly by ivy resolves I think?

Can we have the new types/classes be package private to scalding? They seem like an internal implementation detail as far as scalding is concerned, would be nice to not have them in our ABI if we could? thoughts @johnynek ?

ianoc avatar Feb 13 '16 00:02 ianoc

Re cascading3, I could cherry pick the sbt update commit (moves from Build.scala to build.sbt) to cascading3 branch, and apply this on top of it. If we don't merge this to develop, it will be one more thing to look at in the diff in the eventual cascading3 -> develop merge.

Yes given the pre-existing sub-projects will still pull in classes in the corresponding new *-cascading ones, the upgrade should be fine I think. I'm in favor of merging to develop (and cherry picking to cascading3 branch) if this seems reasonable as a general thing to do.

rubanm avatar Feb 13 '16 00:02 rubanm

Some of Schemes are consumed at Twitter in multiformat sources, for example. The rest could be made package private I think.

rubanm avatar Feb 13 '16 00:02 rubanm

@rubanm we will need help from you and @isnotinvain to make sure we keep twitter up to date and not let any forks happen!

So, if you think you can get the merge in at Twitter, I'm fine with a merge for 0.16.

johnynek avatar Feb 13 '16 01:02 johnynek

Yep, good way to put it. Merge when green from me if the hassle level is ok for you guys to handle to merge.

On Friday, February 12, 2016, P. Oscar Boykin [email protected] wrote:

@rubanm https://github.com/rubanm we will need help from you and @isnotinvain https://github.com/isnotinvain to make sure we keep twitter up to date and not let any forks happen!

So, if you think you can get the merge in at Twitter, I'm fine with a merge for 0.16.

— Reply to this email directly or view it on GitHub https://github.com/twitter/scalding/pull/1514#issuecomment-183558818.

Sent from Gmail Mobile

ianoc avatar Feb 13 '16 04:02 ianoc

@ianoc @johnynek Makes sense. I'll wait for a green sandbox internally before merging. (So we block this on the new algebird version to release internally at Twitter first, with the related breaking tests fixed.) I'll just cherry pick this to the cascading3 branch meanwhile to keep that going.

rubanm avatar Feb 16 '16 17:02 rubanm

Sorry, closed by mistake.

rubanm avatar Feb 16 '16 17:02 rubanm

should we merge this now? I kind of lost track of it.

johnynek avatar Jun 10 '16 07:06 johnynek

CLA assistant check
All committers have signed the CLA.

CLAassistant avatar Nov 16 '19 00:11 CLAassistant