Enable referencing proto files in NuGet packages

[alexvanboxel]

A common pattern in other languages is that your proto files are not part of your project, especially if you are a client but ofter also when you are a server.

Certainly in enterprises that have their stack build on gRPC. What you see is that the proto's are managed in a common mono repo for all contracts (were talking mono-repo for the proto files only here). What you then often see is that contract repo has a language specific distribution step. This do not create client libraries, they just publish the proto files in a package format idiomatic to the language (an example is Java: they package the proto files in a jar package and publish them to a maven repo, in general a private one of the company).

What I like to see is that the .NET implementation would be able to reference proto in NuGet that it can fetch from a NuGet repository. That will make it easier for enterprises to adapt gRPC in .NET. They only need to add the NuGet distribution step in one place and a project just need to references NuGet as an external dependency.

Let me also stress that this is distributing the proto files and not client libraries. You want to avoid distribution of clients that are build with an older version of proto as the one you are using in your project.

And example how this works in Java (using gradle):

dependencies {
    compile("com.google.protobuf:protobuf-java:3.6.1")
    compile("io.grpc:grpc-protobuf:1.18.0")
    protobuf('io.anemos:protobeam-options-proto:0.0.3-SNAPSHOT')
}

The 3th dependency points to an artifact just containing proto, gradle will make sure that client/server stubs are generate with the same version of gRPC and protobuf as the project is.

[JamesNK]

Your Java example, is it like a csproj PackageReference + Protobuf reference of the contents rolled into one?

[JamesNK]

I think you could do this today:

• Package a nupkg with the proto files as content
• Reference the NuGet package in your client/server, which would then include the content files in your project
• Add a <Protobuf> line to your csproj that references the proto files

[JunTaoLuo]

Considerations:

• How to pass the options to the compiler, for example the option to use more efficient APIs for serialization.

[antonioortizpola]

@JamesNK Could you provide an example? I am trying using my protos from other project in my solution (the idea is latter use nuget), but I can not get it to work.

The idea I think is simple, I have a protos project with the proto files in a folder called "Protos". The files are marked as Content.

Then I add that project as reference to mi server project, i can see the references of the files

And I added the Protobuf line

    <ItemGroup>
        <Protobuf Include="Protos\**\*.proto" GrpcServices="Server" AdditionalImportDirs="Protos\" />
        <Content Include="@(Protobuf)" LinkBase="Protos" />
    </ItemGroup>

But the greet.proto is not detected, what am I missing?

[stueeey]

Technically you could make a targets file which enumerated the nuget references and included any .proto files in a well known folder within the packages i.e. 'proto'. might actually be pretty useful if the grpc package did this

[JamesYangLim]

Import proto files from nuget packages work under this csproj settings with grpc.

csproj

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>netstandard2.0</TargetFramework>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Google.Protobuf" Version="3.6.1" />
    <PackageReference Include="Google.Protobuf.Tools" Version="3.6.1" />
    <PackageReference Include="Grpc" Version="1.22.0" />
    <PackageReference Include="Grpc.Core" Version="1.22.0" />
    <PackageReference Include="Grpc.Tools" Version="1.22.0">
      <PrivateAssets>all</PrivateAssets>
      <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
    </PackageReference>
    <PackageReference Include="NugetPackage1" Version="1.0.0" />
    <PackageReference Include="NugetPackage2" Version="1.0.1" />
  </ItemGroup>

  <ItemGroup>
    <NugetPkg1Path Include="$(NuGetPackageRoot)nugetpackage1\1.0.0" />
    <NugetPkg2Path Include="$(NuGetPackageRoot)nugetpackage1\1.0.1" />
    <Protobuf Include="*.proto" AdditionalImportDirs="@(NugetPkg1Path );@(NugetPkg2Path )" />
  </ItemGroup>

</Project>

proto

syntax = "proto3";
import "Pkg1.proto";
import "Pkg2.proto";
package test;
option csharp_namespace = "ProtoTest";

message M_Test
{
    pkg1.M_Pkg1 p1 = 1;
    pkg2.M_Pkg2 p2 = 2;
}

However, I could not workout how to get the version of the nuget package that I want. Instead, I have to hard code in it.

Any thoughts?

[stueeey]

This should do everything you want:

<Target Name="IncludeNugetProtoFiles" BeforeTargets="PreBuildEvent">
    <ItemGroup>
        <_ProtoNugetSearchPaths Include="$(NuGetPackageRoot)%(PackageReference.Identity)\%(PackageReference.Version)\Proto"/>
        <_ProtoNugetFilesFound Include="%(_ProtoNugetSearchPaths.FullPath)\*.Proto"/>
        <Protobuf Remove="@(_ProtoNugetFilesFound)"/>
        <Protobuf Include="@(_ProtoNugetFilesFound)"/>
    </ItemGroup>
    <PropertyGroup>
        <_FoundProtoFiles Condition="'@(_ProtoNugetFilesFound->Count())' &gt; 1">True</_FoundProtoFiles>
    </PropertyGroup>
    <Message Text="Searched for proto files in:" Importance="Normal" />
    <Message Text="&#009;%(_ProtoNugetSearchPaths.FullPath)" Importance="Normal"/>
    <Message Text="Found @(_ProtoNugetFilesFound->Count()) packaged protos:" Importance="Normal" Condition="'$(_FoundProtoFiles)' == 'True'"/>
    <Message Text="&#009;%(_ProtoNugetFilesFound.Identity)" Importance="Normal" Condition="'$(_FoundProtoFiles)' == 'True'"/>
</Target>

This will only work with PackageReference (the new format) but i think we should just accept this as a limitation as the old packages.config way is deprecated anyway and would be a lot harder to implement.

Output:

I'd suggest changing the importance of the first 2 messages printing the search paths to low once you are done testing but leave the others at normal.

Let me know if you need any more help, I've done a lot of MSBuild magic lately

[stueeey]

Actually it might be worth renaming _ProtoNugetSearchPaths to ProtoSearchPaths because then it can be easily added to like so:

<ItemGroup>
    <ProtoSearchPaths Include="SomePath1;SomePath2"/>
</ItemGroup>

[JamesYangLim]

Thanks @stueeey , that helps a lot. However, I am having another issue which I couldn't reference the protobuf type in C# code, error says the type (proto message type) could not be found.

csproj:

  <Target Name="PreBuild" BeforeTargets="PreBuildEvent">
    <ItemGroup>
      <Protobuf Include="*.proto;" />
    </ItemGroup>
  </Target>

proto:

syntax = "proto3";
package x;
option csharp_namespace = "ProtoTest";

message M_X
{
    double v = 1;
}

Build succeeded, however in C# code:

If I removed the target,

csproj:

  <ItemGroup>
    <Protobuf Include="*.proto;" />
  </ItemGroup>

Build succeeded and in C# code:

Seems like the target element trigger msbuild to do something different, not exactly sure what happen behind the scene of msbuild. But this simple test proof that having under target block does not work for grpc.

[ghost]

@stueeey Did you put that snippet into the csproj of the project that contains the proto file or the one that wants to consume it? I actually need what @JamesYangLim was doing with the AdditionalImportDirs so you can reference the proto file from the nuget package in a proto file in your project. Your snippet does not solve that, does it?

[llawall]

Using GeneratePathProperty on the nuget packge

<PackageReference Include="My.BaseProtos" Version="1.0.0" GeneratePathProperty="true" />

and adding that path as an protobuf include

<Protobuf Include="$(PkgMy_BaseProtos)/*.proto" ... />

seems to solve the problem here?

[JamesYangLim]

Hi @llawall, thank you for helping. Your approach works very well and very simple. I have the following example that works.

  <ItemGroup>
    <PackageReference Include="Google.Protobuf" Version="3.6.1" />
    <PackageReference Include="Google.Protobuf.Tools" Version="3.6.1" />
    <PackageReference Include="Grpc" Version="1.22.0" />
    <PackageReference Include="Grpc.Core" Version="1.22.0" />
    <PackageReference Include="Grpc.Tools" Version="1.22.0">
      <PrivateAssets>all</PrivateAssets>
      <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
    </PackageReference>
    <PackageReference Include="my.package.1" Version="2.3.1" GeneratePathProperty="true"/>
    <PackageReference Include="my.package.2" Version="1.3.4" GeneratePathProperty="true"/>
  </ItemGroup>

  <ItemGroup>
    <Protobuf Include="*.proto" AdditionalImportDirs="$(Pkgmy_package_1)\proto;$(Pkgmy_package_2)\proto" />
  </ItemGroup>

[stueeey]

I had some time today and I've managed to get this working with no changes to the nuget references with the following:

<PropertyGroup>
    <Proto_DefaultAccess>Public</Proto_DefaultAccess>
    <Proto_DefaultClientBaseType>ClientBase</Proto_DefaultClientBaseType>
</PropertyGroup>

<Target Name="_includeNugetProtoFiles" BeforeTargets="_Protobuf_SelectFiles">
    <ItemGroup>
        <_ServerProtoNugetSearchPaths Include="$(NuGetPackageRoot)%(PackageReference.Identity)\%(PackageReference.Version)\Protobuf\Grpc\Server" Condition="Exists('$(NuGetPackageRoot)%(PackageReference.Identity)\%(PackageReference.Version)\Protobuf\Grpc\Server')" />
        <_ServerProtoNugetFilesFound Include="%(_ServerProtoNugetSearchPaths.FullPath)\*.proto" />

        <_ClientProtoNugetSearchPaths Include="$(NuGetPackageRoot)%(PackageReference.Identity)\%(PackageReference.Version)\Protobuf\Grpc\Client" Condition="Exists('$(NuGetPackageRoot)%(PackageReference.Identity)\%(PackageReference.Version)\Protobuf\Grpc\Client')" />
        <_ClientProtoNugetFilesFound Include="%(_ClientProtoNugetSearchPaths.FullPath)\*.proto" />

        <_BothProtoNugetSearchPaths Include="$(NuGetPackageRoot)%(PackageReference.Identity)\%(PackageReference.Version)\Protobuf\Grpc" Condition="Exists('$(NuGetPackageRoot)%(PackageReference.Identity)\%(PackageReference.Version)\Protobuf\Grpc')" />
        <_BothProtoNugetFilesFound Include="%(_BothProtoNugetSearchPaths.FullPath)\*.proto" />

        <_ProtoNugetSearchPaths Include="@(_BothProtoNugetSearchPaths)" />
        <_ProtoNugetSearchPaths Include="@(_ServerProtoNugetSearchPaths)" />
        <_ProtoNugetSearchPaths Include="@(_ClientProtoNugetSearchPaths)" />

        <Protobuf Remove="@(_ServerProtoNugetFilesFound)" />
        <Protobuf Include="@(_ServerProtoNugetFilesFound)" GrpcServices="Server" ClientBaseType="$(Proto_DefaultAccess)" Access="$(Proto_DefaultAccess)" OutputDir="$(Protobuf_OutputPath)\PackagedProtos" />

        <Protobuf Remove="@(_ClientProtoNugetFilesFound)" />
        <Protobuf Include="@(_ClientProtoNugetFilesFound)" GrpcServices="Client" ClientBaseType="$(Proto_DefaultAccess)" Access="$(Proto_DefaultAccess)" OutputDir="$(Protobuf_OutputPath)\PackagedProtos"/>

        <Protobuf Remove="@(_BothProtoNugetFilesFound)" />
        <Protobuf Include="@(_BothProtoNugetFilesFound)" GrpcServices="Both" ClientBaseType="$(Proto_DefaultAccess)" Access="$(Proto_DefaultAccess)" OutputDir="$(Protobuf_OutputPath)\PackagedProtos"/>
    </ItemGroup>
    <PropertyGroup>
        <_NugetProtoSearchPathsFound Condition="'@(_BothProtoNugetSearchPaths-&gt;Count())' &gt;= 1">True</_NugetProtoSearchPathsFound>
        <_FoundClientProtoFiles Condition="'@(_ClientProtoNugetFilesFound-&gt;Count())' &gt;= 1">True</_FoundClientProtoFiles>
        <_FoundServerProtoFiles Condition="'@(_ServerProtoNugetFilesFound-&gt;Count())' &gt;= 1">True</_FoundServerProtoFiles>
        <_FoundBothProtoFiles Condition="'@(_BothProtoNugetFilesFound-&gt;Count())' &gt;= 1">True</_FoundBothProtoFiles>
        <_FoundAnyPackagedProtoFiles Condition="'$(_FoundClientProtoFiles)' == 'True' OR '$(_FoundServerProtoFiles)' == 'True' OR '$(_FoundBothProtoFiles)' == 'True'">True</_FoundAnyPackagedProtoFiles>
    </PropertyGroup>
    <Message Text="Including packaged protos with the following settings:" Condition="'$(_FoundAnyPackagedProtoFiles)' == 'True'"/>
    <Message Text=" Proto_DefaultAccess: $(Proto_DefaultAccess)" Condition="'$(_FoundAnyPackagedProtoFiles)' == 'True'"/>
    <Message Text=" Proto_DefaultClientBaseType: $(Proto_DefaultClientBaseType)" Condition="'$(_FoundAnyPackagedProtoFiles)' == 'True'"/>
    <Message Text="Searched for proto files in:" Importance="Normal" Condition="'$(_NugetProtoSearchPathsFound)' == 'True'" />
    <Message Text="	%(_ProtoNugetSearchPaths.FullPath)" Importance="Normal" Condition="'$(_NugetProtoSearchPathsFound)' == 'True'" />
    <Message Text="Found @(_ClientProtoNugetFilesFound-&gt;Count()) packaged client proto(s):" Importance="Normal" Condition="'$(_FoundClientProtoFiles)' == 'True'" />
    <Message Text="	%(_ClientProtoNugetFilesFound.Identity)" Importance="Normal" Condition="'$(_FoundClientProtoFiles)' == 'True'" />
    <Message Text="Found @(_ServerProtoNugetFilesFound-&gt;Count()) packaged server proto(s):" Importance="Normal" Condition="'$(_FoundServerProtoFiles)' == 'True'" />
    <Message Text="	%(_ServerProtoNugetFilesFound.Identity)" Importance="Normal" Condition="'$(_FoundServerProtoFiles)' == 'True'" />
    <Message Text="Found @(_BothProtoNugetFilesFound-&gt;Count()) packaged client &amp; server proto(s):" Importance="Normal" Condition="'$(_FoundBothProtoFiles)' == 'True'" />
    <Message Text="	%(_BothProtoNugetFilesFound.Identity)" Importance="Normal" Condition="'$(_FoundBothProtoFiles)' == 'True'" />
</Target>

This will pick up anything under <NugetPackage>\protobuf\grpc and generate both client and server <NugetPackage>\protobuf\grpc\client and generate a client <NugetPackage>\protobuf\grpc\server and generate a server

Works like a dream:

Planning to add a property to the properties page of proto files to make it automatically include the proto in this folder structure when packing the project

I'll make a PR into the main GRPC repo - this probably belongs in the GRPC.Tools nuget package

[ghost]

If you want to ship .proto files in a nuget package for reuse in .proto files in your own project you can do this:

<ItemGroup>
  <PackageReference Include="MyCompany.ProtoBuf" Version="1.0" GeneratePathProperty="true" />
</ItemGroup>

<ItemGroup>
  <Protobuf Include="Grpc/*.proto" GrpcServices="Both" CompileOutputs="true" AdditionalImportDirs="$(PkgMyCompany_ProtoBuf)/content/Proto" ProtoRoot="Grpc" />
</ItemGroup>

The key here are the GeneratePathProperty and AdditonalImportDirs attributes.

In the MyCompany.ProtoBuf package you need to have something like this:

<ItemGroup>
    <Content Include="Proto\**\*.proto">
      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
    </Content>
</ItemGroup>

to get the proto files inlcuded.

[dameeks]

I wasn't able to get the AdditionalImportDirs to work. It did not work with a non-nuget proto file either (e.g. the proto was located at c:\temp). I ended up doing this:

  <ItemGroup>
    <Protobuf Include="$(PkgMyCompany_ProtoBuf)\Content\Proto\*.proto" GrpcServices="server" CompileOutputs="true" />
  </ItemGroup>
  <ItemGroup>
    <Protobuf Include="gRPC\**\*.proto" GrpcServices="server" CompileOutputs="true" />
  </ItemGroup>

[ghost]

@dameeks: What does this give you? Don't you get the generated client/server already by just referencing the package. Why do you need to generate it again in the consuming project?

[dameeks]

@roederja2 My NuGet package contains a single proto file. Adding a reference to the NuGet package adds the proto file to my csproj, but it does build a gRPC client/server. The only way I got it to build was to add a second ItemGroup. I also tried using

<Protobuf update="$(PkgMyCompany_ProtoBuf)\Content\Proto\*.proto" >

within the same ItemGroup as the include but that did not work either.

[dmitrydvm]

<ItemGroup>
    <Protobuf Include="*.proto;" />
</ItemGroup>

by @JamesYangLim works fine for me.

[ipzKellyR]

This is very similar to an issue I'm trying to solve. I can't use protos imported into other protos that are linked through a class library.

If I use the AdditionalImportDirs flag as seen it will compile, but then when you actually try to reference the something.proto messages in the org.service.proto messages then it says file not found.

<Protobuf Include="..\Auth0Service.Grpc\Protos\org.service.proto" GrpcServices="Client" AdditionalImportDirs="Protos">         
  <Link>Protos\org.service.proto</Link>
</Protobuf>
<Protobuf Include="..\Auth0Service.Grpc\Protos\something.proto" GrpcServices="Client" ProtoCompile="true" Access="Public" >
  <Link>Protos\something.proto</Link>
**</Protobuf>**

[RichyP7]

Reading the intro more exact would have saved us a lot of time.

Let me also stress that this is distributing the proto files and not client libraries. You want to avoid distribution of clients that are build with an older version of proto as the one you are using in your project.

@roederja2 ´s advice for shipping protos works perfectly. Even we found a way to work around the issue, there should be a solution (like a flag or something similar) to easily import the proto files from a consumed nuget. Maybe the packaging of proto files in a standardized way would help.

[stueeey]

@RichyP7 I've done exactly this, they didn't approve it :/ https://github.com/grpc/grpc/pull/20071

[neilwashere]

Thought I'd weigh in as I'm trying to do exactly this: package proto's in nuget package for redistribution such that models can be used in other projects requiring the proto definitions. Specifically sharing 'model' definitions that will be used to build other services. I can package and distribute but inclusion is simply not working using the above methods. dotnet build will setup content links - I can even see them relative to local proto files in the IDE - but will fail with missing files.

I can include packaged protos locally only if I manually setup a symlink to the packaged proto files. Maybe this is specific to Linux (I don't have a window machine to test on) and therefore a separate issue altogether.

[neilwashere]

I have a solution that I can live with for now but it is by no means ideal. It also assumes running in a *nix env which is fine for my needs. Maybe it can help another soul hitting the same issue (MSbuild links are not honoured when compiling local protos using imports from pkg)

<ItemGroup>
    <PackageReference Include="My.Protos" Version="1.0.0" GeneratePathProperty="true" />
</ItemGroup>
<ItemGroup>
     <Protobuf Include="protos/**/*.proto" />
</ItemGroup>
<PropertyGroup>
    <PreBuildEvent>
        ln -sf $(PkgMy_Protos)/content/Some/Protos $(MSBuildProjectDirectory)/protos/foo
    </PreBuildEvent>
</PropertyGroup>

[stueeey]

@neilwashere if that is all you want to do you can just do this:

<Protobuf Include="$(PkgMy_Protos)/content/Some/Protos/**/*.proto"/>

Includes are cumulative so you can leave your existing Protobuf include where it is if you have local proto files too.

My PR which hasn't gone in would allow you to just put your proto files in /protobuf/grpc in your nuget package and they would be automatically found and included without any MSBuild witchery

[RichyP7]

@RichyP7 I've done exactly this, they didn't approve it :/ grpc/grpc#20071

I really think this would be an huge improvement but maybe it doesn´t fit the current modular .net approach. Maybe a plugin for nuget would be the right approach like in java. maven protocol buffer

[neilwashere]

@stueeey thanks for the reply. Unfortunately in my case, where I need to reference the raw packaged protos in order for my local ones to build, MSBuild just says no. Without an explicit symlink the compiler errors with: File not found and my_local_proto.proto(5,1): import "my/external/referenced/protofile.proto" was not found or had errors :feelsgood: No amount of adjusting the import paths for reference seems to work.

[stueeey]

@neilwashere I can probably give you some msbuild magic for that when I'm home

[emanuel-v-r]

How can it be done when referencing directly another project that contains the proto file? I have this in the project that contains the file:

<ItemGroup>
     <Content Include="Protos\**\*.proto">
               <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
      </Content>
</ItemGroup>

And then I have this in the project that is referencing the project containing the file:

<ItemGroup>
	<Protobuf Include="**/*.proto;"  GrpcServices="Client" />
</ItemGroup>

Still, seems to not get detected.

[tverboon]

This is how I do it. I create a client NuGet package per client. This is what goes in the Client project:

<ItemGroup>
  <Protobuf Include="..\Server.Project\Protos\*.proto" GrpcServices="Client" AdditionalImportDirs="..\Server.Project\" />
</ItemGroup>

[emanuel-v-r]

This is how I do it. I create a client NuGet package per client. This is what goes in the Client project:

<ItemGroup>
  <Protobuf Include="..\Server.Project\Protos\*.proto" GrpcServices="Client" AdditionalImportDirs="..\Server.Project\" />
</ItemGroup>

Ended up with that, thanks!