[WIP]BIGTOP-3785: Improve the build scripts to configure package installation path

[kevinw66]

…ion path

Description of PR

How was this patch tested?

For code changes:

• [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'BIGTOP-3638. Your PR title ...')?
• [ ] Make sure that newly added files do not have any licensing issues. When in doubt refer to https://www.apache.org/licenses/

[kevinw66]

@guyuqi @iwasakims @sekikn Could you give some advice? After that I'll start working on the rest of the components.

[sekikn]

Thank you so much for addressing this, @kevinw66! As the first step, I tried to build ZK, Hive and Kafka without the "prefix" option for regression test, and found a few differences between the original and patched versions in zookeeper-debuginfo-*.rpm, hive-*.rpm, and hive-hcatalog-*.rpm. Are they ignorable?

$ ./gradlew allclean zookeeper-pkg hive-pkg kafka-pkg

...

BUILD SUCCESSFUL in 10m 10s
61 actionable tasks: 60 executed, 1 up-to-date
$ mv output /tmp
$ curl -sL https://github.com/apache/bigtop/pull/986.patch | git apply
$ ./gradlew allclean zookeeper-pkg hive-pkg kafka-pkg

...

BUILD SUCCESSFUL in 10m 31s
61 actionable tasks: 61 executed
$ for i in $(ls output/zookeeper/x86_64/zookeeper-*.rpm output/hive/noarch/hive-*.rpm output/kafka/noarch/kafka-*.rpm); do diff <(rpm2cpio $i | cpio -t) <(rpm2cpio /tmp/$i | cpio -t); done

...

5,6c5,6
< ./etc/hive/conf.dist/hive-exec-log4j2.properties
< ./etc/hive/conf.dist/hive-log4j2.properties
---
> ./etc/hive/conf.dist/hive-exec-log4j2.properties.template
> ./etc/hive/conf.dist/hive-log4j2.properties.template

...

19a20
> ./usr/lib/hive-hcatalog/share/hcatalog/hive-hcatalog-core.jar
20a22
> ./usr/lib/hive-hcatalog/share/hcatalog/hive-hcatalog-pig-adapter.jar
21a24
> ./usr/lib/hive-hcatalog/share/hcatalog/hive-hcatalog-server-extensions.jar
22a26
> ./usr/lib/hive-hcatalog/share/hcatalog/hive-hcatalog-streaming.jar

...

3,5c3,8
< ./usr/lib/debug/.build-id/25
< ./usr/lib/debug/.build-id/25/99a4194db78c707491676ca8a885aaa4b4847b
< ./usr/lib/debug/.build-id/25/99a4194db78c707491676ca8a885aaa4b4847b.debug
---
> ./usr/lib/debug/.build-id/1c
> ./usr/lib/debug/.build-id/1c/2a4bc08aa51c0a0e7046efe024bb300adf178c
> ./usr/lib/debug/.build-id/1c/2a4bc08aa51c0a0e7046efe024bb300adf178c.debug
> ./usr/lib/debug/.build-id/51
> ./usr/lib/debug/.build-id/51/c986cae6ab713722a029b7ac694cfe8d69d4eb
> ./usr/lib/debug/.build-id/51/c986cae6ab713722a029b7ac694cfe8d69d4eb.debug
11,16c14,16
< ./usr/lib/debug/.build-id/b6
< ./usr/lib/debug/.build-id/b6/cfa769954d585b08ae64bd5c63e4c35e01289e
< ./usr/lib/debug/.build-id/b6/cfa769954d585b08ae64bd5c63e4c35e01289e.debug
< ./usr/lib/debug/.build-id/d0
< ./usr/lib/debug/.build-id/d0/955dae110a27a1c642f8a10ef92741bc9d7141
< ./usr/lib/debug/.build-id/d0/955dae110a27a1c642f8a10ef92741bc9d7141.debug
---
> ./usr/lib/debug/.build-id/d8
> ./usr/lib/debug/.build-id/d8/379b9351c595c39bfc3d00613cfbdf84650222
> ./usr/lib/debug/.build-id/d8/379b9351c595c39bfc3d00613cfbdf84650222.debug

[sekikn]

I also ran ./gradlew allclean zookeeper-pkg hive-pkg kafka-pkg -Pprefix=/usr/bigtop, and confirmed that the files under /usr were relocated to $prefix/$bigtop_version/usr, and the ones under /etc and /var were not. It almost looks good, but is it OK that /etc/(zookeeper|hive|kafka) aren't relocated, because configuration files can be incompatible between different versions?

[kevinw66]

install-hive.sh is outdated for build hive-*.rpm, since log4j is replaced with log4j2. I don't know where is the zookeeper-debuginfo comes from, but it seems generated by rpmbuild itself, I think it can be ignored. The difference in hive-hcatalog-* should be a bug, let me check on it.

[kevinw66]

I also ran ./gradlew allclean zookeeper-pkg hive-pkg kafka-pkg -Pprefix=/usr/bigtop, and confirmed that the files under /usr were relocated to $prefix/$bigtop_version/usr, and the ones under /etc and /var were not. It almost looks good, but is it OK that /etc/(zookeeper|hive|kafka) aren't relocated, because configuration files can be incompatible between different versions?

/var/log,/var/run,/etc/%{name}/conf won't be relocated to /usr/bigtop/${version}/, but /var/lib will.

I'm thinking we should create symlink from alternatives confdir to /usr/lib/%{name}/conf, it'll be easier for components' upgrade.

[kevinw66]

@sekikn Fixed

[kevinw66]

install-hive.sh is outdated for build hive-*.rpm, since log4j is replaced with log4j2.

Let me create a separate PR for this

[kevinw66]

I also ran ./gradlew allclean zookeeper-pkg hive-pkg kafka-pkg -Pprefix=/usr/bigtop, and confirmed that the files under /usr were relocated to $prefix/$bigtop_version/usr, and the ones under /etc and /var were not. It almost looks good, but is it OK that /etc/(zookeeper|hive|kafka) aren't relocated, because configuration files can be incompatible between different versions?

/var/log,/var/run,/etc/%{name}/conf won't be relocated to /usr/bigtop/${version}/, but /var/lib will.

I'm thinking we should create symlink from alternatives confdir to /usr/lib/%{name}/conf, it'll be easier for components' upgrade.

I think I can relocate all dir to /usr/bigtop/${version}/, and create symlink for /var/log,/var/run,/etc/%{name}/conf in bigtop-select in the future.

[kevinw66]

Pull requests will be created for each component for easy review