Migrate from sbt-protobuf to sbt-protoc

[RustedBones]

This enables grpc java codegen support,

Clean useless scio-schemas. Move test schemas to project's Test configuration.

[codecov[bot]]

Codecov Report

Merging #4483 (1171aaa) into main (f2d69f9) will decrease coverage by 0.11%. The diff coverage is n/a.

@@            Coverage Diff             @@
##             main    #4483      +/-   ##
==========================================
- Coverage   60.26%   60.14%   -0.12%     
==========================================
  Files         275      274       -1     
  Lines       10046    10014      -32     
  Branches      839      839              
==========================================
- Hits         6054     6023      -31     
+ Misses       3992     3991       -1     

Impacted Files	Coverage Δ
.../com/spotify/scio/parquet/read/ParquetReadFn.scala	84.90% <0.00%> (-0.95%)	:arrow_down:
...c/test/scala/com/spotify/scio/avro/AvroUtils.scala
...y/scio/values/PairSkewedSCollectionFunctions.scala	95.94% <0.00%> (+2.70%)	:arrow_up:

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[RustedBones]

Right, the version needs to be set. Can you share your error when using the newer version ?

[clairemcginty]

Right, the version needs to be set. Can you share your error when using the newer version ?

yes that's what I meant - even with version set (the same way as in 6296bebeb879cc3a87b7b7475bfb735eb75e853f), the plugin still tried to download protoc 3.13.0.

I realized the issue was that by putting PB.protocVersion in protobufSettings, it was only getting picked up by the few sbt subprojects that explicitly include protobufSettings:

scio % sbt protocVersion   
[info] welcome to sbt 1.7.1 (Homebrew Java 11.0.15)
...
[info] scio-cassandra3 / protocVersion
[info] 	3.13.0
[info] scio-jmh / protocVersion
[info] 	3.13.0
[info] scio-core / protocVersion
[info] 	3.13.0
[info] scio-repl / protocVersion
[info] 	3.13.0
[info] scio-elasticsearch6 / protocVersion
[info] 	3.13.0
[info] scio-test / protocVersion
[info] 	3.19.4
[info] scio-macros / protocVersion
[info] 	3.13.0
[info] scio-elasticsearch8 / protocVersion
[info] 	3.13.0
[info] scio-extra / protocVersion
[info] 	3.13.0
[info] scio-tensorflow / protocVersion
[info] 	3.19.4
[info] scio-elasticsearch7 / protocVersion
[info] 	3.13.0
[info] scio-jdbc / protocVersion
[info] 	3.13.0
[info] scio-parquet / protocVersion
[info] 	3.13.0
[info] scio-examples / protocVersion
[info] 	3.13.0
[info] scio-smb / protocVersion
[info] 	3.13.0
[info] scio-google-cloud-platform / protocVersion
[info] 	3.13.0
[info] scio-avro / protocVersion
[info] 	3.13.0
[info] scio-redis / protocVersion
[info] 	3.13.0
[info] protocVersion
[info] 	3.13.0

But sbt-protoc will try to eagerly download protocExecutable when compile is called, for every sbt subproject, and that's where the error comes from. So for scio-test/scio-tensorflow compilation succeeds but every other project fails. I was able to get it working by adding PB.protocVersion setting to commonSettings.

[RustedBones]

Ha ok! I think the trick here is to globally disable all auto-plugins and enable only for the desired projects. Will give this a shot

[RustedBones]

@clairemcginty can you validate this again ?