Developer Guide instructions not working – cannot build neural-search

[intrafindBreno]

What is the bug?

The instructions to start working on the neural-search plugin in the DEVELOPER_GUIDE.md seem to be outdated. Cloning the repo and ./gradlew building it will result in the build failing:

➜  neural-search git:(main) ./gradlew build        
=======================================
OpenSearch Build Hamster says Hello!
  Gradle Version        : 8.4
  OS Info               : Mac OS X 14.2.1 (aarch64)
  JDK Version           : 17 (Eclipse Temurin JDK)
  JAVA_HOME             : /Users/breno/.sdkman/candidates/java/17.0.8-tem
  Random Testing Seed   : 387E3680F23D1663
  In FIPS 140 mode      : false
=======================================

> Task :compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

> Task :integTest FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':integTest'.
> Could not resolve all files for configuration ':opensearch_distro_extracted_testclusters--integTest-0-3.0.0-SNAPSHOT-'.
   > Could not resolve opensearch-distribution-snapshot:opensearch:3.0.0-SNAPSHOT.
     Required by:
         project :
      > Could not resolve opensearch-distribution-snapshot:opensearch:3.0.0-SNAPSHOT.
         > Could not get resource 'https://artifacts.opensearch.org/snapshots/core/opensearch/3.0.0-SNAPSHOT/opensearch-min-3.0.0-SNAPSHOT-darwin-arm64-latest.tar.gz'.
            > Could not HEAD 'https://artifacts.opensearch.org/snapshots/core/opensearch/3.0.0-SNAPSHOT/opensearch-min-3.0.0-SNAPSHOT-darwin-arm64-latest.tar.gz'. Received status code 403 from server: Forbidden

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
> Get more help at https://help.gradle.org.

Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

For more on this, please refer to https://docs.gradle.org/8.4/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.

BUILD FAILED in 3s
31 actionable tasks: 8 executed, 23 up-to-date   

It seems there is an access restriction to https://artifacts.opensearch.org/snapshots/core/opensearch/3.0.0-SNAPSHOT/opensearch-min-3.0.0-SNAPSHOT-darwin-arm64-latest.tar.gz, a requirement to run the ITs

How can one reproduce the bug?

1. Clone the neural-search repository
2. follow the instructions in the DEVELOPER_GUIDE.md
3. run ./gradlew build

What is the expected behavior?

The project builds.

What is your host/environment?

MacOS Sonoma 14.2.1, M2

Do you have any screenshots?

N/A

Do you have any additional context?

No.

[intrafindBreno]

I was able to build the project on a Linux machine. So it probably is just a misleading http error code. I assume it should be a 404?

[navneet1v]

@intrafindBreno are you using M1/M2/M3 mac?

[intrafindBreno]

@navneet1v I’m on an M2 Mac.

[navneet1v]

@intrafindBreno Opensearch doesn't publish artifacts for arm based macos. Hence you are facing this problem.

You can follow this documentation: https://github.com/opensearch-project/k-NN/blob/main/DEVELOPER_GUIDE.md#extra-setup-for-mac-m1-machines and start from this step: Next, obtain a minimum distribution tarball of the k-NN version you want to build:

This can help you run the Neural Search with M1/M2 macs