[BUG] Running integration tests with `customDistributionUrl` has no effect

[0xf4b1]

Describe the bug Running integration tests with customDistributionUrl as described in DEVELOPER_GUIDE.md seems not to work. It never accesses the provided url and seems to always build from source.

To Reproduce Steps to reproduce the behavior:

1. Build distribution: ./gradlew clean :distribution:archives:linux-tar:assemble
2. Put built distribution in local web server
3. Run integration tests: ./gradlew clean integTest -PcustomDistributionUrl="https://localhost/opensearch-min-3.0.0-SNAPSHOT-linux-x64.tar.gz"
4. Check web server's access log

Expected behavior Distribution Download Plugin gets the distribution from web server and runs the tests instead of building from source again.

[dblock]

What do you see in the log and what do you expect?

Would need to be debugged, the code is here.

[0xf4b1]

In the web server's access log, I'm expecting to see an access to the file opensearch-min-3.0.0-SNAPSHOT-linux-x64.tar.gz but it does not occur. Also, if I provide an invalid url or random characters to customDistributionUrl, no error occurs that's why it seems to me to have no effect.

[dblock]

@Rishikesh1159

[saratvemulapalli]

@0xf4b1 thanks for reaching out. Could you try using the absolute path and it should work. Something like: ./gradlew build -PcustomDistributionUrl=/home/ubuntu/OpenSearch/distribution/archives/linux-tar/build/distributions/opensearch-min-3.0.0-SNAPSHOT-linux-x64.tar.gz

[dblock]

If that does work, we should add support for relative paths, better logging to show that the value is used, etc.

[0xf4b1]

@saratvemulapalli thank you for looking into this. Unfortunately, I think this doesn't work either. Since I don't have an access log now, I did a quick test with strace strace -e trace=open,openat,close -f -o trace.log ./gradlew integTest -PcustomDistributionUrl=/home/fabi/opensearch-min-3.0.0-SNAPSHOT-linux-x64.tar.gz and the log reveals it never accesses the file but instead accesses distribution/archives/integ-test-zip/build/distributions/opensearch-min-3.0.0-SNAPSHOT.zip that was rebuilt.

In the meantime, I took a look into the opensearch-build repo and grep'ed for that option, but it's never used there. Maybe this option is not used anywhere and no longer working?

[dblock]

@0xf4b1 it's entirely possible that it's broken - try to debug it, I would throw some System.println's in there and see what it comes up with to start

[Rishikesh1159]

@0xf4b1 sorry for late response, I missed this issue.

After looking your concern I verified that this feature is still working and not broken. I picked Anomaly Detection plugin to test this feature (any plugin using gradle will work).

Verfiication Process:

-> Pick any opensarch plugin which is using gradle. I picked Anomaly Detection for testing. -> Clone the plugin repo. Run any gradle command with -PcustomDistributionUrl . Example: ./gradlew integTest -PcustomDistributionUrl="https://ci.opensearch.org/ci/dbc/bundle-build/1.2.0/1127/linux/x64/dist/opensearch-1.2.0-linux-x64.tar.gz" -> I also verified by passing an invalid url and it throws an error as expected.

:~/Pictures/anomaly-detection$ ./gradlew integTest -PcustomDistributionUrl="invalid url"
=======================================
OpenSearch Build Hamster says Hello!
  Gradle Version        : 7.4.2
  OS Info               : Linux 5.15.0-1019-aws (amd64)
  JDK Version           : 17 (OpenJDK)
  JAVA_HOME             : /opt/jdk-17
  Random Testing Seed   : 83988B3332F1CC
  In FIPS 140 mode      : false
=======================================

> Task :compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

> Task :compileTestJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

> Task :integTest FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':integTest'.
> Could not resolve all files for configuration ':opensearch_distro_extracted_testclusters--integTest-0-2.3.0-SNAPSHOT-'.
   > Could not find opensearch-distribution-snapshot:opensearch:2.3.0-SNAPSHOT.
     Searched in the following locations:
       - file:/home/rrpasham/Pictures/anomaly-detection/invalid url
     Required by:
         project :

-> I also verified by passing tar.gz from local machine and it worked fine. We need to put our tar.gz in plugin folder for "customDistributionUrl" to pick it. Example : -PcustomDistributionUrl="opensearch-min-2.2.0-linux-x64.tar.gz"

If possible can please you provide more details on your failure @0xf4b1 ? Like what plugins are using to test this and error you get. Just to be clear on correct usage/purpose of "customDistributionUrl" I stated it below:

Usage/Purpose of customDistributionUrl :

-> Main purpose this was introduced is for testing on opensearch plugins. Opensearch plugins sometimes needs to test/run against different versions of opensearch. In that case this customDistributionUrl is very useful to test on unreleased/old versions of opensearch. -> -PcustomDistributionUrl="" is always set from the plugins end. Opensearch repo itself don't use -PcustomDistributionUrl="" only opensearch plugins can set this.

[Rishikesh1159]

Closing this. Please free to reopen @0xf4b1 if you have any more questions.