protoc-gen-grpc-java not available on apple m1

[reinmind]

What version of gRPC-Java are you using?

1.34.0

What is your environment?

osx 11.0.1

What did you expect to see?

build success

What did you see instead?

[INFO] --- protobuf-maven-plugin:0.6.1:compile (default-cli) @ grpc ---
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  0.720 s
[INFO] Finished at: 2020-12-03T16:37:02+08:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.xolstice.maven.plugins:protobuf-maven-plugin:0.6.1:compile (default-cli) on project grpc: Unable to resolve artifact: Missing:
[ERROR] ----------
[ERROR] 1) com.google.protobuf:protoc:exe:osx-aarch_64:3.12.0
[ERROR] 
[ERROR]   Try downloading the file manually from the project website.
[ERROR] 
[ERROR]   Then, install it using the command: 
[ERROR]       mvn install:install-file -DgroupId=com.google.protobuf -DartifactId=protoc -Dversion=3.12.0 -Dclassifier=osx-aarch_64 -Dpackaging=exe -Dfile=/path/to/file
[ERROR] 
[ERROR]   Alternatively, if you host your own repository you can deploy the file there: 
[ERROR]       mvn deploy:deploy-file -DgroupId=com.google.protobuf -DartifactId=protoc -Dversion=3.12.0 -Dclassifier=osx-aarch_64 -Dpackaging=exe -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id]
[ERROR] 
[ERROR]   Path to dependency: 
[ERROR]   	1) com.reinmind:grpc:jar:1.0-SNAPSHOT
[ERROR]   	2) com.google.protobuf:protoc:exe:osx-aarch_64:3.12.0
[ERROR] 
[ERROR] ----------
[ERROR] 1 required artifact is missing.
[ERROR] 
[ERROR] for artifact: 
[ERROR]   com.reinmind:grpc:jar:1.0-SNAPSHOT
[ERROR] 
[ERROR] from the specified remote repositories:
[ERROR]   aliyunmaven (https://maven.aliyun.com/repository/public, releases=true, snapshots=false)
[ERROR] 
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

Steps to reproduce the bug

mvn protobuf:compile

[ejona86]

Yes, we do not produce binaries for arm64 macs.

---

Edit: Summary of the long issue: The easiest thing to do as a consumer is to install Rosetta. With it installed, the Intel binary works fine. If you'd like to contribute, see https://github.com/grpc/grpc-java/issues/7690#issuecomment-831617279 . Easiest plan is to build protobuf and the grpc plugin as universal binaries.

[ejona86]

Since this is the grpc repo, I've called out protoc-gen-grpc-java not being available. Although your actual error is protoc itself, which would be part of the protobuf project. If grpc gets to building it first, then we'd need to add support in protobuf as well.

[gaurav46]

Hi @ejona86 Is it possible for any workaround like to set osdetector.arch as x86_64 to get the complication done ? I tried to set below in gradle.properties but it still looks for osx-aarch_64 binaries from maven. org.gradle.jvmargs=-Dos.detected.name=osx -Dos.detected.arch=x86_64 -Dos.detected.classifier=osx-x86_64

[ejona86]

To manually specify the classifier, for Gradle:

protobuf {
  protoc {
    artifact = 'com.google.protobuf:protoc:3.14.0:osx-x86_64'
  }
}

I'm less familiar with the Maven plugin, but I think it may be something like:

<protocArtifact>com.google.protobuf:protoc:3.14.0:exe:osx-x86_64</protocArtifact>

You can do the same for protoc-gen-grpc-java.

[gaurav46]

Thanks @ejona86 . That worked out.

[zjc17]

Thanks @ejona86 , But How can I keep other platform like linux-x86_64 when the project being compiled on other platform?

[ejona86]

@Jiachen-Zhang

protobuf {
  protoc {
    if (osdetector.os == "osx") {
      artifact = 'com.google.protobuf:protoc:3.14.0:osx-x86_64'
    } else {
      artifact = 'com.google.protobuf:protoc:3.14.0'
    }
  }
}

You can also use project properties to only change the behavior when explicitly requested.

[zjc17]

@Jiachen-Zhang

protobuf {
  protoc {
    if (osdetector.os == "osx") {
      artifact = 'com.google.protobuf:protoc:3.14.0:osx-x86_64'
    } else {
      artifact = 'com.google.protobuf:protoc:3.14.0'
    }
  }
}

You can also use project properties to only change the behavior when explicitly requested.

Thanks!

Finally, I made it by

// for apple m1, please add protoc_platform=osx-x86_64 in $HOME/.gradle/gradle.properties
if (project.hasProperty('protoc_platform')) {
    artifact = "com.google.protobuf:protoc:3.13.0:${protoc_platform}"
} else {
    artifact = "com.google.protobuf:protoc:3.13.0"
}

[hu-chia]

for me, add this to ~/.m2/settings.xml, it worked:

<settings>
  ...
  <activeProfiles>
    <activeProfile>
      apple-silicon
    </activeProfile>
    ...
  </activeProfiles>
  <profiles>
    <profile>
      <id>apple-silicon</id>
      <properties>
        <os.detected.classifier>osx-x86_64</os.detected.classifier>
      </properties>
    </profile>
    ...
  </profiles>
  ...
</settings>

[Uditmittal]

Can you share the full setting.xml I tried it didn't work for me? I am using M1 with protobuf.

[zjc17]

@Uditmittal in my build.gradle file

protobuf {
    protoc {
        // for apple m1, please add protoc_platform=osx-x86_64 in $HOME/.gradle/gradle.properties
        if (project.hasProperty('protoc_platform')) {
            artifact = "com.google.protobuf:protoc:3.13.0:${protoc_platform}"
        } else {
            artifact = "com.google.protobuf:protoc:3.13.0"
        }
    }
}

in my ~/.gradle/gradle.properties file

protoc_platform=osx-x86_64

I think you can define and use variables in maven to implement this feature. Good luck

[hu-chia]

@Uditmittal My settings.xml is awfully long, the key is using your property overwrite kr.motd.maven:os-maven-plugin detected os classifier, if you share your settings.xml and pom.xml snippet, this may be help.

[Uditmittal]

<settings>
  <activeProfiles>
    <activeProfile>
      apple-silicon
    </activeProfile>
  </activeProfiles>
  <profiles>
    <profile>
      <id>apple-silicon</id>
      <properties>
        <os.detected.classifier>osx-x86_64</os.detected.classifier>
      </properties>
    </profile>
  </profiles>
</settings>

@guyeu my seetings.xml file

[hu-chia]

@Uditmittal This is my org.xolstice.maven.plugins:protobuf-maven-plugin build plugin configuration, the ${os.detected.classifier} value could read from active profile from settings.xml.

      <configuration>
          <protocArtifact>
              com.google.protobuf:protoc:3.1.0:exe:${os.detected.classifier}
          </protocArtifact>
          <pluginId>grpc-java</pluginId>
          <pluginArtifact>io.grpc:protoc-gen-grpc-java:1.0.0:exe:${os.detected.classifier}</pluginArtifact>
      </configuration>

[Uditmittal]

Thanks, it works for me.

[bangbang2333]

谢谢老哥，搞定了

[ejona86]

I'm not at all impressed with how hard it was to find the appropriate flags to pass to clang to make a universal binary, but it appears to be -arch arm64 -arch x86_64. Unclear if cross-compiling is as simple as -arch arm64. We can start by figuring out how to cross-compile/universal binary for protobuf. Afterward we can fight Gradle. The goal would be to let us build M1 binaries from an x86 Mac. But I don't have a Mac which means I can't help much, as develop-via-CI is a form of torture.

[ghost]

@ejona86 Have you had any success building from source for AArch64? I am interested in helping moving this along...

[ejona86]

@jjones-figure, no, as I said, I don't have a Mac, so I can't really help much here. We totally build from source for AArch64 already on Linux. But it is a different toolchain for M1.

[ejona86]

https://github.com/protocolbuffers/protobuf/pull/8557 just went into protobuf based on my suggestion to just copy the current x86 binary to an arm64 name until we have actual M1 support. That doesn't "fix" this, but it does resolve the main user-visible issues and doesn't turn out to be too hacky.

I think that is as simple as making a copy of the "exe" and the hashes/signature in our upload_artifacts.sh script.

[kdubb]

@ejona86 Where do I beg to get protoc-gen-grpc-java published as osx-aarch_64?

This is the lone holdout of 3 projects (including com.google.protobuf:protoc) required for building easily on the M1. The other projects seem to be following the advice of copying osx-x86_64 to osx-aarch_64 and it works well. I currently do this manually on my machine to get the required protoc-gen-grpc-java:osx-aarch_64 artifact.

[kdubb]

@ejona86 I've had a look at the repo and created a very simple PR #8680 . No idea if it works but maybe this gets things rolling?

[liuxuliangcumt]

@Uditmittal in my build.gradle file

protobuf {
    protoc {
        // for apple m1, please add protoc_platform=osx-x86_64 in $HOME/.gradle/gradle.properties
        if (project.hasProperty('protoc_platform')) {
            artifact = "com.google.protobuf:protoc:3.13.0:${protoc_platform}"
        } else {
            artifact = "com.google.protobuf:protoc:3.13.0"
        }
    }
}

in my ~/.gradle/gradle.properties file

protoc_platform=osx-x86_64

I think you can define and use variables in maven to implement this feature. Good luck

in flutter_blue android. build.gradle add protobuf... you say in my build.gradle file .how could we know in' flutter_blue android build.gradle 'to add ?

[solohsu]

https://gist.github.com/solohsu/a2bac0498ef51c105cedd2dc0a854bcf I write a more general Gradle workaround script which may be helpful to you guys who come across any plugin that doesn't support macOS M1 Soc, not only protobuf.

[liuxuliangcumt]

谢谢您的来信！

[ilya-yarets]

To manually specify the classifier, for Gradle:

protobuf {
  protoc {
    artifact = 'com.google.protobuf:protoc:3.14.0:osx-x86_64'
  }
}

I'm less familiar with the Maven plugin, but I think it may be something like:

<protocArtifact>com.google.protobuf:protoc:3.14.0:exe:osx-x86_64</protocArtifact>

You can do the same for protoc-gen-grpc-java.

thank you very much! it's working

[ejona86]

Reopening because we've only a workaround currently. We still need to do something along the lines of https://github.com/grpc/grpc-java/issues/7690#issuecomment-831617279 for native M1 support. I thought we had another issue open for tracking native M1 support, but I can't find it. Anyway, this will do.

[liuxuliangcumt]

谢谢您的来信！

[sukhbir-singh]

Is this solved ? I am still facing this issue.

[ejona86]

Interactions with the protobuf plugins is resolved, but requires Rosetta. I reopened the issue to track native M1 support.