incubator-uniffle icon indicating copy to clipboard operation
incubator-uniffle copied to clipboard

Make spark3 as activation profile

Open hddong opened this issue 3 years ago • 8 comments

What changes were proposed in this pull request?

Make spark3 as activation profile

Why are the changes needed?

Idea can not resolved dependencies if there are no activation profile

Cannot resolve org.apache.spark:spark-core_${scala.binary.version}:${spark.version}

Does this PR introduce any user-facing change?

No

How was this patch tested?

test locally

hddong avatar Jul 08 '22 06:07 hddong

Actually we can manually set active profiles in IDEA like this,

image

So this change is not necessary, and the solution is limited to spark3 profile.

kaijchen avatar Jul 08 '22 06:07 kaijchen

Codecov Report

Merging #39 (64eb319) into master (8256765) will increase coverage by 0.54%. The diff coverage is n/a.

@@             Coverage Diff              @@
##             master      #39      +/-   ##
============================================
+ Coverage     56.29%   56.83%   +0.54%     
- Complexity     1168     1204      +36     
============================================
  Files           152      152              
  Lines          8401     8401              
  Branches        813      813              
============================================
+ Hits           4729     4775      +46     
+ Misses         3408     3368      -40     
+ Partials        264      258       -6     
Impacted Files Coverage Δ
...a/org/apache/uniffle/common/RemoteStorageInfo.java 94.44% <0.00%> (+31.48%) :arrow_up:
.../java/org/apache/uniffle/common/BufferSegment.java 100.00% <0.00%> (+44.00%) :arrow_up:
...java/org/apache/uniffle/common/PartitionRange.java 100.00% <0.00%> (+100.00%) :arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 8256765...64eb319. Read the comment docs.

codecov-commenter avatar Jul 08 '22 06:07 codecov-commenter

I'm neutral for activeByDefault, but recommend to unify the profile style, e.g.

spark-2.3
spark-2.4
spark-3.0
spark-3.1
spark-3.2

pan3793 avatar Jul 08 '22 07:07 pan3793

I'm neutral for activeByDefault, but recommend to unify the profile style, e.g.

spark-2.3
spark-2.4
spark-3.0
spark-3.1
spark-3.2

Agreed.

kaijchen avatar Jul 08 '22 07:07 kaijchen

yes, the profile name should have same style

colinmjj avatar Jul 08 '22 07:07 colinmjj

I'm neutral for activeByDefault, but recommend to unify the profile style, e.g.

spark-2.3
spark-2.4
spark-3.0
spark-3.1
spark-3.2

It's good to unify the style. But I want to have a default profile for Spark2 and Spark3, the default profile is the version which we mainly support. So we make the some profile names special. Could you give some suggestion to solve this problem?

jerqi avatar Jul 08 '22 07:07 jerqi

... the default profile is the version which we mainly support.

With the community growing, other versions of Spark may be adopted by users in large-scale production, how about pointing the version on the use case sharing page(something like Who is using Uniffle)?

pan3793 avatar Jul 11 '22 01:07 pan3793

... the default profile is the version which we mainly support.

With the community growing, other versions of Spark may be adopted by users in large-scale production, how about pointing the version on the use case sharing page(something like Who is using Uniffle)?

Good idea.

jerqi avatar Jul 11 '22 03:07 jerqi