druid icon indicating copy to clipboard operation
druid copied to clipboard
verified publisher icon apache

Upgraded Hadoop to 3.4.0

Open naanagon opened this issue 1 year ago • 12 comments

Upgraded Hadoop to 3.4.0

naanagon avatar Oct 09 '24 15:10 naanagon

@naanagon - have you done a Hadoop Ingestion with this change? Can you also get a diff of what jars changed in the distribution tar as a result of this upgrade?

abhishekagarwal87 avatar Oct 10 '24 05:10 abhishekagarwal87 

diff main_old/dep.tree upgraded/dep.tree
918,920c918,920
< [INFO] +- org.apache.hadoop:hadoop-client-api:jar:3.3.6:provided
< [INFO] +- org.apache.hadoop:hadoop-client-runtime:jar:3.3.6:provided
< [INFO] +- org.apache.hadoop:hadoop-client-minicluster:jar:3.3.6:test
---
> [INFO] +- org.apache.hadoop:hadoop-client-api:jar:3.4.0:provided
> [INFO] +- org.apache.hadoop:hadoop-client-runtime:jar:3.4.0:provided
> [INFO] +- org.apache.hadoop:hadoop-client-minicluster:jar:3.4.0:test
1133c1133
< [INFO] \- org.apache.hadoop:hadoop-client-api:jar:3.3.6:provided
---
> [INFO] \- org.apache.hadoop:hadoop-client-api:jar:3.4.0:provided
1803c1803
< [INFO] +- org.apache.hadoop:hadoop-client-api:jar:3.3.6:compile
---
> [INFO] +- org.apache.hadoop:hadoop-client-api:jar:3.4.0:compile
1806c1806
< [INFO] \- org.apache.hadoop:hadoop-client-runtime:jar:3.3.6:runtime
---
> [INFO] \- org.apache.hadoop:hadoop-client-runtime:jar:3.4.0:runtime
1943c1943
< [INFO] +- org.apache.hadoop:hadoop-client-api:jar:3.3.6:compile
---
> [INFO] +- org.apache.hadoop:hadoop-client-api:jar:3.4.0:compile
1947c1947
< [INFO] \- org.apache.hadoop:hadoop-client-runtime:jar:3.3.6:test
---
> [INFO] \- org.apache.hadoop:hadoop-client-runtime:jar:3.4.0:test
2171,2172c2171,2172
< [INFO] +- org.apache.hadoop:hadoop-client-api:jar:3.3.6:compile
< [INFO] \- org.apache.hadoop:hadoop-client-runtime:jar:3.3.6:runtime
---
> [INFO] +- org.apache.hadoop:hadoop-client-api:jar:3.4.0:compile
> [INFO] \- org.apache.hadoop:hadoop-client-runtime:jar:3.4.0:runtime
3288c3288,3289
< [INFO] +- org.apache.hadoop:hadoop-aws:jar:3.3.6:runtime
---
> [INFO] +- org.apache.hadoop:hadoop-aws:jar:3.4.0:runtime
> [INFO] |  +- software.amazon.awssdk:bundle:jar:2.23.19:runtime
3391,3392c3392,3393
< [INFO] +- org.apache.hadoop:hadoop-client-api:jar:3.3.6:compile
< [INFO] +- org.apache.hadoop:hadoop-client-runtime:jar:3.3.6:runtime
---
> [INFO] +- org.apache.hadoop:hadoop-client-api:jar:3.4.0:compile
> [INFO] +- org.apache.hadoop:hadoop-client-runtime:jar:3.4.0:runtime
3394c3395
< [INFO] +- org.apache.hadoop:hadoop-client-minicluster:jar:3.3.6:test
---
> [INFO] +- org.apache.hadoop:hadoop-client-minicluster:jar:3.4.0:test
5414,5415c5415,5416
< [INFO] |  +- org.apache.hadoop:hadoop-client-api:jar:3.3.6:provided
< [INFO] |  \- org.apache.hadoop:hadoop-client-runtime:jar:3.3.6:provided
---
> [INFO] |  +- org.apache.hadoop:hadoop-client-api:jar:3.4.0:provided
> [INFO] |  \- org.apache.hadoop:hadoop-client-runtime:jar:3.4.0:provided
5526c5527,5528
< [INFO] |  \- org.apache.hadoop:hadoop-aws:jar:3.3.6:provided
---
> [INFO] |  \- org.apache.hadoop:hadoop-aws:jar:3.4.0:provided
> [INFO] |     +- software.amazon.awssdk:bundle:jar:2.23.19:provided
6749,6750c6751,6752
< [INFO] |  +- io.netty:netty-common:jar:4.1.108.Final:provided
< [INFO] |  +- io.netty:netty-buffer:jar:4.1.108.Final:provided
---
> [INFO] |  +- io.netty:netty-common:jar:4.1.108.Final:compile
> [INFO] |  +- io.netty:netty-buffer:jar:4.1.108.Final:compile
6752,6753c6754,6755
< [INFO] |  |  +- io.netty:netty-transport:jar:4.1.108.Final:provided
< [INFO] |  |  \- io.netty:netty-codec:jar:4.1.108.Final:provided
---
> [INFO] |  |  +- io.netty:netty-transport:jar:4.1.108.Final:compile
> [INFO] |  |  \- io.netty:netty-codec:jar:4.1.108.Final:compile
6784d6785
< [INFO] |  |  |  \- io.netty:netty-transport-classes-epoll:jar:4.1.108.Final:provided
6856,6858c6857,6859
< [INFO] |  +- io.netty:netty-handler:jar:4.1.108.Final:provided
< [INFO] |  |  +- io.netty:netty-resolver:jar:4.1.108.Final:provided
< [INFO] |  |  \- io.netty:netty-transport-native-unix-common:jar:4.1.108.Final:provided
---
> [INFO] |  +- io.netty:netty-handler:jar:4.1.108.Final:compile
> [INFO] |  |  +- io.netty:netty-resolver:jar:4.1.108.Final:compile
> [INFO] |  |  \- io.netty:netty-transport-native-unix-common:jar:4.1.108.Final:compile
6867,6869c6868,6870
< [INFO] +- org.apache.hadoop:hadoop-common:jar:3.3.6:compile
< [INFO] |  +- org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_7:jar:1.1.1:compile
< [INFO] |  +- org.apache.hadoop.thirdparty:hadoop-shaded-guava:jar:1.1.1:compile
---
> [INFO] +- org.apache.hadoop:hadoop-common:jar:3.4.0:compile
> [INFO] |  +- org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_21:jar:1.2.0:compile
> [INFO] |  +- org.apache.hadoop.thirdparty:hadoop-shaded-guava:jar:1.2.0:compile
6871a6873
> [INFO] |  +- org.codehaus.jettison:jettison:jar:1.5.4:compile
6876a6879,6880
> [INFO] |  +- io.netty:netty-transport-native-epoll:jar:4.1.108.Final:compile
> [INFO] |  |  \- io.netty:netty-transport-classes-epoll:jar:4.1.108.Final:compile
6878,6881c6882,6886
< [INFO] |  +- org.apache.kerby:kerb-core:jar:1.0.1:compile
< [INFO] |  |  \- org.apache.kerby:kerby-pkix:jar:1.0.1:compile
< [INFO] |  |     +- org.apache.kerby:kerby-asn1:jar:1.0.1:compile
< [INFO] |  |     \- org.apache.kerby:kerby-util:jar:1.0.1:compile
---
> [INFO] |  +- org.bouncycastle:bcprov-jdk15on:jar:1.70:compile
> [INFO] |  +- org.apache.kerby:kerb-core:jar:2.0.3:compile
> [INFO] |  |  \- org.apache.kerby:kerby-pkix:jar:2.0.3:compile
> [INFO] |  |     +- org.apache.kerby:kerby-asn1:jar:2.0.3:compile
> [INFO] |  |     \- org.apache.kerby:kerby-util:jar:2.0.3:compile
6884c6889
< [INFO] |  \- dnsjava:dnsjava:jar:2.1.7:compile
---
> [INFO] |  \- dnsjava:dnsjava:jar:3.4.0:compile
6904,6915c6909,6922
< [INFO] +- org.apache.hadoop:hadoop-auth:jar:3.3.6:compile
< [INFO] |  \- org.apache.kerby:kerb-simplekdc:jar:1.0.1:compile
< [INFO] |     +- org.apache.kerby:kerb-client:jar:1.0.1:compile
< [INFO] |     |  +- org.apache.kerby:kerby-config:jar:1.0.1:compile
< [INFO] |     |  +- org.apache.kerby:kerb-common:jar:1.0.1:compile
< [INFO] |     |  |  \- org.apache.kerby:kerb-crypto:jar:1.0.1:compile
< [INFO] |     |  +- org.apache.kerby:kerb-util:jar:1.0.1:compile
< [INFO] |     |  \- org.apache.kerby:token-provider:jar:1.0.1:compile
< [INFO] |     \- org.apache.kerby:kerb-admin:jar:1.0.1:compile
< [INFO] |        +- org.apache.kerby:kerb-server:jar:1.0.1:compile
< [INFO] |        |  \- org.apache.kerby:kerb-identity:jar:1.0.1:compile
< [INFO] |        \- org.apache.kerby:kerby-xdr:jar:1.0.1:compile
---
> [INFO] +- org.apache.hadoop:hadoop-auth:jar:3.4.0:compile
> [INFO] |  \- org.apache.kerby:kerb-simplekdc:jar:2.0.3:compile
> [INFO] |     +- org.apache.kerby:kerb-client:jar:2.0.3:compile
> [INFO] |     |  +- org.apache.kerby:kerby-config:jar:2.0.3:compile
> [INFO] |     |  +- org.apache.kerby:kerb-common:jar:2.0.3:compile
> [INFO] |     |  |  \- org.apache.kerby:kerb-crypto:jar:2.0.3:compile
> [INFO] |     |  +- org.apache.kerby:kerb-util:jar:2.0.3:compile
> [INFO] |     |  \- org.apache.kerby:token-provider:jar:2.0.3:compile
> [INFO] |     \- org.apache.kerby:kerb-admin:jar:2.0.3:compile
> [INFO] |        +- org.apache.kerby:kerb-server:jar:2.0.3:compile
> [INFO] |        |  \- org.apache.kerby:kerb-identity:jar:2.0.3:compile
> [INFO] |        +- org.apache.kerby:kerby-xdr:jar:2.0.3:compile
> [INFO] |        +- org.jline:jline:jar:3.22.0:compile
> [INFO] |        \- com.jcraft:jsch:jar:0.1.55:compile
8311,8312c8318,8319
< [INFO] +- org.apache.hadoop:hadoop-client-api:jar:3.3.6:compile
< [INFO] \- org.apache.hadoop:hadoop-client-runtime:jar:3.3.6:runtime
---
> [INFO] +- org.apache.hadoop:hadoop-client-api:jar:3.4.0:compile
> [INFO] \- org.apache.hadoop:hadoop-client-runtime:jar:3.4.0:runtime
10447c10454
< [INFO] +- org.apache.hadoop:hadoop-client-api:jar:3.3.6:compile
---
> [INFO] +- org.apache.hadoop:hadoop-client-api:jar:3.4.0:compile
10449c10456
< [INFO] \- org.apache.hadoop:hadoop-client-runtime:jar:3.3.6:runtime
---
> [INFO] \- org.apache.hadoop:hadoop-client-runtime:jar:3.4.0:runtime
10632c10639
< [INFO] +- org.apache.hadoop:hadoop-annotations:jar:3.3.6:compile
---
> [INFO] +- org.apache.hadoop:hadoop-annotations:jar:3.4.0:compile
14016,14019c14023,14026
< [INFO] +- org.apache.hadoop:hadoop-common:jar:3.3.6:compile
< [INFO] |  +- org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_7:jar:1.1.1:compile
< [INFO] |  +- org.apache.hadoop:hadoop-annotations:jar:3.3.6:compile
< [INFO] |  +- org.apache.hadoop.thirdparty:hadoop-shaded-guava:jar:1.1.1:compile
---
> [INFO] +- org.apache.hadoop:hadoop-common:jar:3.4.0:compile
> [INFO] |  +- org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_21:jar:1.2.0:compile
> [INFO] |  +- org.apache.hadoop:hadoop-annotations:jar:3.4.0:compile
> [INFO] |  +- org.apache.hadoop.thirdparty:hadoop-shaded-guava:jar:1.2.0:compile
14021a14029
> [INFO] |  +- org.codehaus.jettison:jettison:jar:1.5.4:compile
14026,14037c14034,14053
< [INFO] |  +- org.apache.hadoop:hadoop-auth:jar:3.3.6:compile
< [INFO] |  |  \- org.apache.kerby:kerb-simplekdc:jar:1.0.1:compile
< [INFO] |  |     +- org.apache.kerby:kerb-client:jar:1.0.1:compile
< [INFO] |  |     |  +- org.apache.kerby:kerby-config:jar:1.0.1:compile
< [INFO] |  |     |  +- org.apache.kerby:kerb-common:jar:1.0.1:compile
< [INFO] |  |     |  |  \- org.apache.kerby:kerb-crypto:jar:1.0.1:compile
< [INFO] |  |     |  +- org.apache.kerby:kerb-util:jar:1.0.1:compile
< [INFO] |  |     |  \- org.apache.kerby:token-provider:jar:1.0.1:compile
< [INFO] |  |     \- org.apache.kerby:kerb-admin:jar:1.0.1:compile
< [INFO] |  |        +- org.apache.kerby:kerb-server:jar:1.0.1:compile
< [INFO] |  |        |  \- org.apache.kerby:kerb-identity:jar:1.0.1:compile
< [INFO] |  |        \- org.apache.kerby:kerby-xdr:jar:1.0.1:compile
---
> [INFO] |  +- org.apache.hadoop:hadoop-auth:jar:3.4.0:compile
> [INFO] |  |  \- org.apache.kerby:kerb-simplekdc:jar:2.0.3:compile
> [INFO] |  |     +- org.apache.kerby:kerb-client:jar:2.0.3:compile
> [INFO] |  |     |  +- org.apache.kerby:kerby-config:jar:2.0.3:compile
> [INFO] |  |     |  +- org.apache.kerby:kerb-common:jar:2.0.3:compile
> [INFO] |  |     |  |  \- org.apache.kerby:kerb-crypto:jar:2.0.3:compile
> [INFO] |  |     |  +- org.apache.kerby:kerb-util:jar:2.0.3:compile
> [INFO] |  |     |  \- org.apache.kerby:token-provider:jar:2.0.3:compile
> [INFO] |  |     \- org.apache.kerby:kerb-admin:jar:2.0.3:compile
> [INFO] |  |        +- org.apache.kerby:kerb-server:jar:2.0.3:compile
> [INFO] |  |        |  \- org.apache.kerby:kerb-identity:jar:2.0.3:compile
> [INFO] |  |        +- org.apache.kerby:kerby-xdr:jar:2.0.3:compile
> [INFO] |  |        \- org.jline:jline:jar:3.22.0:compile
> [INFO] |  +- io.netty:netty-handler:jar:4.1.108.Final:compile
> [INFO] |  |  +- io.netty:netty-resolver:jar:4.1.108.Final:compile
> [INFO] |  |  +- io.netty:netty-transport:jar:4.1.108.Final:compile
> [INFO] |  |  +- io.netty:netty-transport-native-unix-common:jar:4.1.108.Final:compile
> [INFO] |  |  \- io.netty:netty-codec:jar:4.1.108.Final:compile
> [INFO] |  +- io.netty:netty-transport-native-epoll:jar:4.1.108.Final:compile
> [INFO] |  |  \- io.netty:netty-transport-classes-epoll:jar:4.1.108.Final:compile
14039,14042c14055,14059
< [INFO] |  +- org.apache.kerby:kerb-core:jar:1.0.1:compile
< [INFO] |  |  \- org.apache.kerby:kerby-pkix:jar:1.0.1:compile
< [INFO] |  |     +- org.apache.kerby:kerby-asn1:jar:1.0.1:compile
< [INFO] |  |     \- org.apache.kerby:kerby-util:jar:1.0.1:compile
---
> [INFO] |  +- org.bouncycastle:bcprov-jdk15on:jar:1.70:compile
> [INFO] |  +- org.apache.kerby:kerb-core:jar:2.0.3:compile
> [INFO] |  |  \- org.apache.kerby:kerby-pkix:jar:2.0.3:compile
> [INFO] |  |     +- org.apache.kerby:kerby-asn1:jar:2.0.3:compile
> [INFO] |  |     \- org.apache.kerby:kerby-util:jar:2.0.3:compile
14116,14123c14133,14135
< [INFO] |  +- io.netty:netty-common:jar:4.1.108.Final:provided
< [INFO] |  +- io.netty:netty-buffer:jar:4.1.108.Final:provided
< [INFO] |  +- io.netty:netty-codec-http:jar:4.1.108.Final:provided
< [INFO] |  |  +- io.netty:netty-transport:jar:4.1.108.Final:provided
< [INFO] |  |  |  \- io.netty:netty-resolver:jar:4.1.108.Final:provided
< [INFO] |  |  +- io.netty:netty-codec:jar:4.1.108.Final:provided
< [INFO] |  |  \- io.netty:netty-handler:jar:4.1.108.Final:provided
< [INFO] |  |     \- io.netty:netty-transport-native-unix-common:jar:4.1.108.Final:provided
---
> [INFO] |  +- io.netty:netty-common:jar:4.1.108.Final:compile
> [INFO] |  +- io.netty:netty-buffer:jar:4.1.108.Final:runtime
> [INFO] |  +- io.netty:netty-codec-http:jar:4.1.108.Final:runtime
14151,14156c14163,14166
< [INFO] |  |  +- io.netty:netty-codec-socks:jar:4.1.108.Final:provided
< [INFO] |  |  +- io.netty:netty-handler-proxy:jar:4.1.108.Final:provided
< [INFO] |  |  +- io.netty:netty-transport-native-epoll:jar:linux-x86_64:4.1.108.Final:provided
< [INFO] |  |  |  \- io.netty:netty-transport-classes-epoll:jar:4.1.108.Final:provided
< [INFO] |  |  +- io.netty:netty-resolver-dns:jar:4.1.108.Final:provided
< [INFO] |  |  |  \- io.netty:netty-codec-dns:jar:4.1.108.Final:provided
---
> [INFO] |  |  +- io.netty:netty-codec-socks:jar:4.1.108.Final:runtime
> [INFO] |  |  +- io.netty:netty-handler-proxy:jar:4.1.108.Final:runtime
> [INFO] |  |  +- io.netty:netty-transport-native-epoll:jar:linux-x86_64:4.1.108.Final:runtime
> [INFO] |  |  +- io.netty:netty-resolver-dns:jar:4.1.108.Final:runtime
14165,14169c14175,14200
< [INFO] +- org.apache.hadoop:hadoop-hdfs-client:jar:3.3.6:runtime
< [INFO] |  \- com.squareup.okhttp3:okhttp:jar:4.9.3:runtime
< [INFO] |     \- com.squareup.okio:okio:jar:2.8.0:runtime
< [INFO] +- org.apache.hadoop:hadoop-mapreduce-client-core:jar:3.3.6:runtime
< [INFO] +- org.apache.hadoop:hadoop-aws:jar:3.3.6:runtime
---
> [INFO] +- org.apache.hadoop:hadoop-hdfs-client:jar:3.4.0:runtime
> [INFO] +- org.apache.hadoop:hadoop-mapreduce-client-core:jar:3.4.0:runtime
> [INFO] |  \- io.netty:netty-all:jar:4.1.108.Final:runtime
> [INFO] |     +- io.netty:netty-codec-dns:jar:4.1.108.Final:runtime
> [INFO] |     +- io.netty:netty-codec-haproxy:jar:4.1.108.Final:runtime
> [INFO] |     +- io.netty:netty-codec-http2:jar:4.1.108.Final:runtime
> [INFO] |     +- io.netty:netty-codec-memcache:jar:4.1.108.Final:runtime
> [INFO] |     +- io.netty:netty-codec-mqtt:jar:4.1.108.Final:runtime
> [INFO] |     +- io.netty:netty-codec-redis:jar:4.1.108.Final:runtime
> [INFO] |     +- io.netty:netty-codec-smtp:jar:4.1.108.Final:runtime
> [INFO] |     +- io.netty:netty-codec-stomp:jar:4.1.108.Final:runtime
> [INFO] |     +- io.netty:netty-codec-xml:jar:4.1.108.Final:runtime
> [INFO] |     +- io.netty:netty-handler-ssl-ocsp:jar:4.1.108.Final:runtime
> [INFO] |     +- io.netty:netty-transport-rxtx:jar:4.1.108.Final:runtime
> [INFO] |     +- io.netty:netty-transport-sctp:jar:4.1.108.Final:runtime
> [INFO] |     +- io.netty:netty-transport-udt:jar:4.1.108.Final:runtime
> [INFO] |     +- io.netty:netty-transport-classes-kqueue:jar:4.1.108.Final:runtime
> [INFO] |     +- io.netty:netty-resolver-dns-classes-macos:jar:4.1.108.Final:runtime
> [INFO] |     +- io.netty:netty-transport-native-epoll:jar:linux-aarch_64:4.1.108.Final:runtime
> [INFO] |     +- io.netty:netty-transport-native-epoll:jar:linux-riscv64:4.1.108.Final:runtime
> [INFO] |     +- io.netty:netty-transport-native-kqueue:jar:osx-x86_64:4.1.108.Final:runtime
> [INFO] |     +- io.netty:netty-transport-native-kqueue:jar:osx-aarch_64:4.1.108.Final:runtime
> [INFO] |     +- io.netty:netty-resolver-dns-native-macos:jar:osx-x86_64:4.1.108.Final:runtime
> [INFO] |     \- io.netty:netty-resolver-dns-native-macos:jar:osx-aarch_64:4.1.108.Final:runtime
> [INFO] +- org.apache.hadoop:hadoop-aws:jar:3.4.0:runtime
> [INFO] |  +- software.amazon.awssdk:bundle:jar:2.23.19:runtime
14210,14211d14240
< [INFO] |  +- org.apache.hadoop:hadoop-client-runtime:jar:3.3.4:compile
< [INFO] |  |  \- commons-logging:commons-logging:jar:1.1.1:runtime
14220,14221c14249,14252
< [INFO] +- org.apache.hadoop:hadoop-client-api:jar:3.3.6:compile
< [INFO] |  \- org.xerial.snappy:snappy-java:jar:1.1.10.4:compile
---
> [INFO] +- org.apache.hadoop:hadoop-client-runtime:jar:3.4.0:compile
> [INFO] |  +- org.xerial.snappy:snappy-java:jar:1.1.10.4:compile
> [INFO] |  \- commons-logging:commons-logging:jar:1.1.1:runtime
> [INFO] +- org.apache.hadoop:hadoop-client-api:jar:3.4.0:compile
14320c14351,14352
< [INFO] +- org.apache.hadoop:hadoop-aws:jar:3.3.6:runtime
---
> [INFO] +- org.apache.hadoop:hadoop-aws:jar:3.4.0:runtime
> [INFO] |  +- software.amazon.awssdk:bundle:jar:2.23.19:runtime
15474c15506,15507
< [INFO] |  +- org.apache.hadoop:hadoop-aws:jar:3.3.6:provided
---
> [INFO] |  +- org.apache.hadoop:hadoop-aws:jar:3.4.0:provided
> [INFO] |  |  +- software.amazon.awssdk:bundle:jar:2.23.19:provided
15476,15477c15509,15510
< [INFO] |  +- org.apache.hadoop:hadoop-client-api:jar:3.3.6:provided
< [INFO] |  \- org.apache.hadoop:hadoop-client-runtime:jar:3.3.6:provided
---
> [INFO] |  +- org.apache.hadoop:hadoop-client-api:jar:3.4.0:provided
> [INFO] |  \- org.apache.hadoop:hadoop-client-runtime:jar:3.4.0:provided

List of new dependencies added:

hadoop-common: org.codehaus.jettison:jettison:jar:1.5.4:compile org.bouncycastle:bcprov-jdk15on:jar:1.70:compile -> https://mvnrepository.com/artifact/org.bouncycastle/bcprov-jdk15on artifact moved to jdk18. Should i go ahead by adding 18 and excluding 15..?

io.netty scope changed

hadoop-auth: org.jline:jline:jar:3.22.0:compile com.jcraft:jsch:jar:0.1.55:compile -> com.jcraft is already excluded from hadoop-common , Should i also exclude from hadoop-auth..?

hadoop-aws: software.amazon.awssdk:bundle:jar:2.23.19:runtime

I have to add licenses to org.codehaus.jettison, com.jcraft -> org.jline.

naanagon avatar Oct 10 '24 09:10 naanagon

I usually build the distribution tar via this command

mvn clean install  -Pdist   -Dgpg.skip -DskipTests -Dforbiddenapis.skip=true -Dcheckstyle.skip=true -Dpmd.skip=true -Dmaven.javadoc.skip=true -Danimal.sniffer.skip=true -Denforcer.skip=true -Dspotbugs.skip=true -Dcyclonedx.skip=true -T1C

The tar would be in distribution/target folder. You can run this command with and without the patch and recursive diff the contents of two tars.

abhishekagarwal87 avatar Oct 10 '24 09:10 abhishekagarwal87

From distribution tar: tar -tzf apache-druid-32.0.0-SNAPSHOT-bin.tar.gz_upgrade | grep '.jar$' | sort > new_jars.txt

➜  compare diff main_old/old_jars.txt upgraded/new_jars.txt
36c36
< apache-druid-32.0.0-SNAPSHOT/extensions/ambari-metrics-emitter/hadoop-annotations-3.3.6.jar
---
> apache-druid-32.0.0-SNAPSHOT/extensions/ambari-metrics-emitter/hadoop-annotations-3.4.0.jar
57c57
< apache-druid-32.0.0-SNAPSHOT/extensions/druid-avro-extensions/hadoop-client-api-3.3.6.jar
---
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-avro-extensions/hadoop-client-api-3.4.0.jar
211a212
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-deltalake-extensions/bundle-2.23.19.jar
218,220c219,221
< apache-druid-32.0.0-SNAPSHOT/extensions/druid-deltalake-extensions/hadoop-aws-3.3.6.jar
< apache-druid-32.0.0-SNAPSHOT/extensions/druid-deltalake-extensions/hadoop-client-api-3.3.6.jar
< apache-druid-32.0.0-SNAPSHOT/extensions/druid-deltalake-extensions/hadoop-client-runtime-3.3.4.jar
---
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-deltalake-extensions/hadoop-aws-3.4.0.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-deltalake-extensions/hadoop-client-api-3.4.0.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-deltalake-extensions/hadoop-client-runtime-3.4.0.jar
237a239
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-hdfs-storage/bundle-2.23.19.jar
241,243c243,245
< apache-druid-32.0.0-SNAPSHOT/extensions/druid-hdfs-storage/hadoop-aws-3.3.6.jar
< apache-druid-32.0.0-SNAPSHOT/extensions/druid-hdfs-storage/hadoop-client-api-3.3.6.jar
< apache-druid-32.0.0-SNAPSHOT/extensions/druid-hdfs-storage/hadoop-client-runtime-3.3.6.jar
---
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-hdfs-storage/hadoop-aws-3.4.0.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-hdfs-storage/hadoop-client-api-3.4.0.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-hdfs-storage/hadoop-client-runtime-3.4.0.jar
256a259,260
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/bcprov-jdk15on-1.70.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/bundle-2.23.19.jar
272,279c276,283
< apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/hadoop-annotations-3.3.6.jar
< apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/hadoop-auth-3.3.6.jar
< apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/hadoop-aws-3.3.6.jar
< apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/hadoop-common-3.3.6.jar
< apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/hadoop-hdfs-client-3.3.6.jar
< apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/hadoop-mapreduce-client-core-3.3.6.jar
< apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/hadoop-shaded-guava-1.1.1.jar
< apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/hadoop-shaded-protobuf_3_7-1.1.1.jar
---
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/hadoop-annotations-3.4.0.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/hadoop-auth-3.4.0.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/hadoop-aws-3.4.0.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/hadoop-common-3.4.0.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/hadoop-hdfs-client-3.4.0.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/hadoop-mapreduce-client-core-3.4.0.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/hadoop-shaded-guava-1.2.0.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/hadoop-shaded-protobuf_3_21-1.2.0.jar
303a308,309
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/jettison-1.5.4.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/jline-3.22.0.jar
307,320c313,326
< apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/kerb-admin-1.0.1.jar
< apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/kerb-client-1.0.1.jar
< apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/kerb-common-1.0.1.jar
< apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/kerb-core-1.0.1.jar
< apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/kerb-crypto-1.0.1.jar
< apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/kerb-identity-1.0.1.jar
< apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/kerb-server-1.0.1.jar
< apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/kerb-simplekdc-1.0.1.jar
< apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/kerb-util-1.0.1.jar
< apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/kerby-asn1-1.0.1.jar
< apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/kerby-config-1.0.1.jar
< apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/kerby-pkix-1.0.1.jar
< apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/kerby-util-1.0.1.jar
< apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/kerby-xdr-1.0.1.jar
---
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/kerb-admin-2.0.3.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/kerb-client-2.0.3.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/kerb-common-2.0.3.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/kerb-core-2.0.3.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/kerb-crypto-2.0.3.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/kerb-identity-2.0.3.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/kerb-server-2.0.3.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/kerb-simplekdc-2.0.3.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/kerb-util-2.0.3.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/kerby-asn1-2.0.3.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/kerby-config-2.0.3.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/kerby-pkix-2.0.3.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/kerby-util-2.0.3.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/kerby-xdr-2.0.3.jar
323,324c329,364
< apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/okhttp-4.9.3.jar
< apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/okio-2.8.0.jar
---
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/netty-all-4.1.108.Final.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/netty-buffer-4.1.108.Final.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/netty-codec-4.1.108.Final.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/netty-codec-dns-4.1.108.Final.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/netty-codec-haproxy-4.1.108.Final.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/netty-codec-http-4.1.108.Final.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/netty-codec-http2-4.1.108.Final.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/netty-codec-memcache-4.1.108.Final.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/netty-codec-mqtt-4.1.108.Final.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/netty-codec-redis-4.1.108.Final.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/netty-codec-smtp-4.1.108.Final.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/netty-codec-socks-4.1.108.Final.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/netty-codec-stomp-4.1.108.Final.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/netty-codec-xml-4.1.108.Final.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/netty-common-4.1.108.Final.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/netty-handler-4.1.108.Final.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/netty-handler-proxy-4.1.108.Final.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/netty-handler-ssl-ocsp-4.1.108.Final.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/netty-resolver-4.1.108.Final.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/netty-resolver-dns-4.1.108.Final.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/netty-resolver-dns-classes-macos-4.1.108.Final.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/netty-resolver-dns-native-macos-4.1.108.Final-osx-aarch_64.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/netty-resolver-dns-native-macos-4.1.108.Final-osx-x86_64.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/netty-transport-4.1.108.Final.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/netty-transport-classes-epoll-4.1.108.Final.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/netty-transport-classes-kqueue-4.1.108.Final.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/netty-transport-native-epoll-4.1.108.Final-linux-aarch_64.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/netty-transport-native-epoll-4.1.108.Final-linux-riscv64.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/netty-transport-native-epoll-4.1.108.Final-linux-x86_64.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/netty-transport-native-epoll-4.1.108.Final.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/netty-transport-native-kqueue-4.1.108.Final-osx-aarch_64.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/netty-transport-native-kqueue-4.1.108.Final-osx-x86_64.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/netty-transport-native-unix-common-4.1.108.Final.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/netty-transport-rxtx-4.1.108.Final.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/netty-transport-sctp-4.1.108.Final.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/netty-transport-udt-4.1.108.Final.jar
329c369
< apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/token-provider-1.0.1.jar
---
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-iceberg-extensions/token-provider-2.0.3.jar
358a399
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-kerberos/bcprov-jdk15on-1.70.jar
362c403
< apache-druid-32.0.0-SNAPSHOT/extensions/druid-kerberos/dnsjava-2.1.7.jar
---
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-kerberos/dnsjava-3.4.0.jar
364,367c405,408
< apache-druid-32.0.0-SNAPSHOT/extensions/druid-kerberos/hadoop-auth-3.3.6.jar
< apache-druid-32.0.0-SNAPSHOT/extensions/druid-kerberos/hadoop-common-3.3.6.jar
< apache-druid-32.0.0-SNAPSHOT/extensions/druid-kerberos/hadoop-shaded-guava-1.1.1.jar
< apache-druid-32.0.0-SNAPSHOT/extensions/druid-kerberos/hadoop-shaded-protobuf_3_7-1.1.1.jar
---
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-kerberos/hadoop-auth-3.4.0.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-kerberos/hadoop-common-3.4.0.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-kerberos/hadoop-shaded-guava-1.2.0.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-kerberos/hadoop-shaded-protobuf_3_21-1.2.0.jar
370a412
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-kerberos/jettison-1.5.4.jar
377a420,421
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-kerberos/jline-3.22.0.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-kerberos/jsch-0.1.55.jar
379,392c423,436
< apache-druid-32.0.0-SNAPSHOT/extensions/druid-kerberos/kerb-admin-1.0.1.jar
< apache-druid-32.0.0-SNAPSHOT/extensions/druid-kerberos/kerb-client-1.0.1.jar
< apache-druid-32.0.0-SNAPSHOT/extensions/druid-kerberos/kerb-common-1.0.1.jar
< apache-druid-32.0.0-SNAPSHOT/extensions/druid-kerberos/kerb-core-1.0.1.jar
< apache-druid-32.0.0-SNAPSHOT/extensions/druid-kerberos/kerb-crypto-1.0.1.jar
< apache-druid-32.0.0-SNAPSHOT/extensions/druid-kerberos/kerb-identity-1.0.1.jar
< apache-druid-32.0.0-SNAPSHOT/extensions/druid-kerberos/kerb-server-1.0.1.jar
< apache-druid-32.0.0-SNAPSHOT/extensions/druid-kerberos/kerb-simplekdc-1.0.1.jar
< apache-druid-32.0.0-SNAPSHOT/extensions/druid-kerberos/kerb-util-1.0.1.jar
< apache-druid-32.0.0-SNAPSHOT/extensions/druid-kerberos/kerby-asn1-1.0.1.jar
< apache-druid-32.0.0-SNAPSHOT/extensions/druid-kerberos/kerby-config-1.0.1.jar
< apache-druid-32.0.0-SNAPSHOT/extensions/druid-kerberos/kerby-pkix-1.0.1.jar
< apache-druid-32.0.0-SNAPSHOT/extensions/druid-kerberos/kerby-util-1.0.1.jar
< apache-druid-32.0.0-SNAPSHOT/extensions/druid-kerberos/kerby-xdr-1.0.1.jar
---
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-kerberos/kerb-admin-2.0.3.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-kerberos/kerb-client-2.0.3.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-kerberos/kerb-common-2.0.3.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-kerberos/kerb-core-2.0.3.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-kerberos/kerb-crypto-2.0.3.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-kerberos/kerb-identity-2.0.3.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-kerberos/kerb-server-2.0.3.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-kerberos/kerb-simplekdc-2.0.3.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-kerberos/kerb-util-2.0.3.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-kerberos/kerby-asn1-2.0.3.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-kerberos/kerby-config-2.0.3.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-kerberos/kerby-pkix-2.0.3.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-kerberos/kerby-util-2.0.3.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-kerberos/kerby-xdr-2.0.3.jar
393a438,446
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-kerberos/netty-buffer-4.1.108.Final.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-kerberos/netty-codec-4.1.108.Final.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-kerberos/netty-common-4.1.108.Final.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-kerberos/netty-handler-4.1.108.Final.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-kerberos/netty-resolver-4.1.108.Final.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-kerberos/netty-transport-4.1.108.Final.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-kerberos/netty-transport-classes-epoll-4.1.108.Final.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-kerberos/netty-transport-native-epoll-4.1.108.Final.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-kerberos/netty-transport-native-unix-common-4.1.108.Final.jar
399c452
< apache-druid-32.0.0-SNAPSHOT/extensions/druid-kerberos/token-provider-1.0.1.jar
---
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-kerberos/token-provider-2.0.3.jar
477,478c530,531
< apache-druid-32.0.0-SNAPSHOT/extensions/druid-orc-extensions/hadoop-client-api-3.3.6.jar
< apache-druid-32.0.0-SNAPSHOT/extensions/druid-orc-extensions/hadoop-client-runtime-3.3.6.jar
---
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-orc-extensions/hadoop-client-api-3.4.0.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-orc-extensions/hadoop-client-runtime-3.4.0.jar
511,512c564,565
< apache-druid-32.0.0-SNAPSHOT/extensions/druid-parquet-extensions/hadoop-client-api-3.3.6.jar
< apache-druid-32.0.0-SNAPSHOT/extensions/druid-parquet-extensions/hadoop-client-runtime-3.3.6.jar
---
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-parquet-extensions/hadoop-client-api-3.4.0.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-parquet-extensions/hadoop-client-runtime-3.4.0.jar
554,555c607,608
< apache-druid-32.0.0-SNAPSHOT/extensions/druid-ranger-security/hadoop-client-api-3.3.6.jar
< apache-druid-32.0.0-SNAPSHOT/extensions/druid-ranger-security/hadoop-client-runtime-3.3.6.jar
---
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-ranger-security/hadoop-client-api-3.4.0.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-ranger-security/hadoop-client-runtime-3.4.0.jar
631,632c684,685
< apache-druid-32.0.0-SNAPSHOT/extensions/druid-thrift-extensions/hadoop-client-api-3.3.6.jar
< apache-druid-32.0.0-SNAPSHOT/extensions/druid-thrift-extensions/hadoop-client-runtime-3.3.6.jar
---
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-thrift-extensions/hadoop-client-api-3.4.0.jar
> apache-druid-32.0.0-SNAPSHOT/extensions/druid-thrift-extensions/hadoop-client-runtime-3.4.0.jar
714a768,776
> apache-druid-32.0.0-SNAPSHOT/hadoop-dependencies/hadoop-client-api/3.4.0/hadoop-client-api-3.4.0.jar
> apache-druid-32.0.0-SNAPSHOT/hadoop-dependencies/hadoop-client-api/3.4.0/snappy-java-1.1.10.4.jar
> apache-druid-32.0.0-SNAPSHOT/hadoop-dependencies/hadoop-client-runtime/3.4.0/commons-logging-1.2.jar
> apache-druid-32.0.0-SNAPSHOT/hadoop-dependencies/hadoop-client-runtime/3.4.0/hadoop-client-api-3.4.0.jar
> apache-druid-32.0.0-SNAPSHOT/hadoop-dependencies/hadoop-client-runtime/3.4.0/hadoop-client-runtime-3.4.0.jar
> apache-druid-32.0.0-SNAPSHOT/hadoop-dependencies/hadoop-client-runtime/3.4.0/jsr305-3.0.2.jar
> apache-druid-32.0.0-SNAPSHOT/hadoop-dependencies/hadoop-client-runtime/3.4.0/reload4j-1.2.22.jar
> apache-druid-32.0.0-SNAPSHOT/hadoop-dependencies/hadoop-client-runtime/3.4.0/slf4j-api-1.7.36.jar
> apache-druid-32.0.0-SNAPSHOT/hadoop-dependencies/hadoop-client-runtime/3.4.0/snappy-java-1.1.10.4.jar

naanagon avatar Oct 10 '24 09:10 naanagon

Most of it seems fine to me except

  • what is in the bundle jar
  • Iceberg extension is pulling in a lot of netty related dependencies. Where do they come from?

abhishekagarwal87 avatar Oct 10 '24 10:10 abhishekagarwal87

  1. Bundle Jar is of hadoop-aws
[INFO] +- org.apache.hadoop:hadoop-aws:jar:3.4.0:runtime
[INFO] |  +- software.amazon.awssdk:bundle:jar:2.23.19:runtime
  1. hadoop-mapreduce-client-core has all the new netty related dependencies
[INFO] +- org.apache.hadoop:hadoop-mapreduce-client-core:jar:3.4.0:runtime
[INFO] |  \- io.netty:netty-all:jar:4.1.108.Final:runtime
[INFO] |     +- io.netty:netty-codec-dns:jar:4.1.108.Final:runtime
[INFO] |     +- io.netty:netty-codec-haproxy:jar:4.1.108.Final:runtime
[INFO] |     +- io.netty:netty-codec-http2:jar:4.1.108.Final:runtime
[INFO] |     +- io.netty:netty-codec-memcache:jar:4.1.108.Final:runtime
[INFO] |     +- io.netty:netty-codec-mqtt:jar:4.1.108.Final:runtime
[INFO] |     +- io.netty:netty-codec-redis:jar:4.1.108.Final:runtime
[INFO] |     +- io.netty:netty-codec-smtp:jar:4.1.108.Final:runtime
[INFO] |     +- io.netty:netty-codec-stomp:jar:4.1.108.Final:runtime
[INFO] |     +- io.netty:netty-codec-xml:jar:4.1.108.Final:runtime
[INFO] |     +- io.netty:netty-handler-ssl-ocsp:jar:4.1.108.Final:runtime
[INFO] |     +- io.netty:netty-transport-rxtx:jar:4.1.108.Final:runtime
[INFO] |     +- io.netty:netty-transport-sctp:jar:4.1.108.Final:runtime
[INFO] |     +- io.netty:netty-transport-udt:jar:4.1.108.Final:runtime
[INFO] |     +- io.netty:netty-transport-classes-kqueue:jar:4.1.108.Final:runtime
[INFO] |     +- io.netty:netty-resolver-dns-classes-macos:jar:4.1.108.Final:runtime
[INFO] |     +- io.netty:netty-transport-native-epoll:jar:linux-aarch_64:4.1.108.Final:runtime
[INFO] |     +- io.netty:netty-transport-native-epoll:jar:linux-riscv64:4.1.108.Final:runtime
[INFO] |     +- io.netty:netty-transport-native-kqueue:jar:osx-x86_64:4.1.108.Final:runtime
[INFO] |     +- io.netty:netty-transport-native-kqueue:jar:osx-aarch_64:4.1.108.Final:runtime
[INFO] |     +- io.netty:netty-resolver-dns-native-macos:jar:osx-x86_64:4.1.108.Final:runtime
[INFO] |     \- io.netty:netty-resolver-dns-native-macos:jar:osx-aarch_64:4.1.108.Final:runtime

naanagon avatar Oct 10 '24 10:10 naanagon

hadoop-common has a new dependency org.bouncycastle:bcprov-jdk15on:jar:1.70:compile -> this version-jdk 1.5 moved to jdk1.8 and also it has CVEs. It is fixed in hadoop 3.4.1 (not released yet) by moving to new jdk1.8.

naanagon avatar Oct 10 '24 10:10 naanagon

Thanks @naanagon for taking this up. Great to see you contributing to Apache Druid 🚀 .

We would most likely move to a LTS stable version of the JDK which is 11/17 https://en.wikipedia.org/wiki/Java_version_history

Also how does the upgrade interact with aws sdk v1 deps which currently druid is using extensively.

cryptoe avatar Oct 11 '24 08:10 cryptoe

@naanagon - have you done a Hadoop Ingestion with this change? Can you also get a diff of what jars changed in the distribution tar as a result of this upgrade?

Tested hadoop ingestion and it is working fine.

naanagon avatar Oct 11 '24 14:10 naanagon

Thanks @naanagon for taking this up. Great to see you contributing to Apache Druid 🚀 .

We would most likely move to a LTS stable version of the JDK is 11/17 https://en.wikipedia.org/wiki/Java_version_history

Also how does the upgrade interact with aws sdk v1 deps which currently druid is using extensively.

Sorry with the confusion of versions, I updated my comment.

I mentioned it to bring to your attention that there is a CVE for bouncycastle. Also, hadoop didn't fix the bouncycastle CVE in 3.4 as the upgrade has few test cases failure.

This upgrade retained all aws sdk v1 dependencies and introduced sdk v2. In terms of changes they added adapters for v1 to interact to v2 link

naanagon avatar Oct 11 '24 14:10 naanagon

@naanagon - FYI we usually don't force push commits since that makes it harder to review incremental changes.

abhishekagarwal87 avatar Oct 15 '24 06:10 abhishekagarwal87

@a2l007 - does the Iceberg extension require dependency to mapreduce-client-core? Can we exclude it?

abhishekagarwal87 avatar Oct 15 '24 07:10 abhishekagarwal87