protobuf
protobuf copied to clipboard
protocolbuffers
How to learn to use protoc in 21+ easily infuriating steps
I was trying to generate C# code from a very small .proto file. Here is my console log. In reality it was twice as long due to duplicate commands issued during this stochastic goal search. To be hones, I'm appalled at the quality of the program command-line interface.
>protoc --version
libprotoc 3.2.0
>protoc
Missing input file.
>protoc a:\proto_dir\CNTK.proto
Missing output directives.
>protoc a:\proto_dir\CNTK.proto --csharp_out A:\Output directory\cntk.proto.cs
a:\proto_dir\CNTK.proto: File does not reside within any path specified using --proto_path (or -I). You must specify a --proto_path which encompasses this file. Note that the proto_path must be an exact prefix of the .proto file names -- protoc is too dumb to figure out when two paths (e.g. absolute and relative) are equivalent (it's harder than you think).
>protoc a:\proto_dir\CNTK.proto --csharp_out A:\Output directory\cntk.proto.cs --proto_path a:\proto_dir\
directory: No such file or directory
>protoc a:\proto_dir\CNTK.proto --csharp_out "A:\Output directory\cntk.proto.cs" --proto_path a:\proto_dir\
A:\Output directory\cntk.proto.cs/: No such file or directory
>protoc a:\proto_dir\CNTK.proto --csharp_out "A:\Output directory\" --proto_path a:\proto_dir\
a:\proto_dir\CNTK.proto: File does not reside within any path specified using --proto_path (or -I). You must specify a --proto_path which encompasses this file. Note that the proto_path must be an exact prefix of the .proto file names -- protoc is too dumb to figure out when two paths (e.g. absolute and relative) are equivalent (it's harder than you think).
>protoc CNTK.proto --csharp_out "A:\Output directory\" --proto_path a:\proto_dir\
CNTK.proto: No such file or directory
>protoc a:\proto_dir\CNTK.proto --csharp_out "A:\Output directory\" --proto_path a:\proto_dir
a:\proto_dir\CNTK.proto: File does not reside within any path specified using --proto_path (or -I). You must specify a --proto_path which encompasses this file. Note that the proto_path must be an exact prefix of the .proto file names -- protoc is too dumb to figure out when two paths (e.g. absolute and relative) are equivalent (it's harder than you think).
>protoc --csharp_out "A:\Output directory\" --proto_path a:\proto_dir\ a:\proto_dir\CNTK.proto
Missing input file.
>protoc --csharp_out "A:\Output directory\" --proto_path a:\proto_dir\ CNTK.proto
Missing input file.
>protoc --csharp_out "A:\Output directory\CNTK.proto.cs" a:\proto_dir\CNTK.proto
a:\proto_dir\CNTK.proto: File does not reside within any path specified using --proto_path (or -I). You must specify a --proto_path which encompasses this file. Note that the proto_path must be an exact prefix of the .proto file names -- protoc is too dumb to figure out when two paths (e.g. absolute and relative) are equivalent (it's harder than you think).
>protoc --csharp_out "A:\Output directory\CNTK.proto.cs" --proto_path=a:\proto_dir\ a:\proto_dir\CNTK.proto
A:\Output directory\CNTK.proto.cs/: No such file or directory
>protoc --csharp_out "A:\Output directory\" --proto_path=a:\proto_dir\ a:\proto_dir\CNTK.proto
Missing input file.
>protoc --csharp_out="A:\Output directory\" --proto_path="a:\proto_dir\" a:\proto_dir\CNTK.proto
a:\proto_dir\CNTK.proto: File does not reside within any path specified using --proto_path (or -I). You must specify a --proto_path which encompasses this file. Note that the proto_path must be an exact prefix of the .proto file names -- protoc is too dumb to figure out when two paths (e.g. absolute and relative) are equivalent (it's harder than you think).
>protoc --csharp_out="A:\Output directory\" --proto_path="a:\proto_dir\" "a:\proto_dir\CNTK.proto"
a:\proto_dir\CNTK.proto: File does not reside within any path specified using --proto_path (or -I). You must specify a --proto_path which encompasses this file. Note that the proto_path must be an exact prefix of the .proto file names -- protoc is too dumb to figure out when two paths (e.g. absolute and relative) are equivalent (it's harder than you think).
>protoc --csharp_out="A:\Output directory\" --proto_path="a:\proto_dir" "a:\proto_dir\CNTK.proto"
Missing input file.
>cd A:\proto_dir
A:\proto_dir>protoc.exe --csharp_out="A:\Output directory\" --proto_path="a:\proto_dir" CNTK.proto
Missing input file.
A:\proto_dir>protoc.exe --csharp_out="A:\Output directory\" --proto_path=. CNTK.proto
Missing input file.
A:\proto_dir>protoc.exe --csharp_out="A:\Output directory\" --proto_path=./ CNTK.proto
Missing input file.
A:\proto_dir>protoc.exe --csharp_out=. CNTK.proto
A:\proto_dir>protoc.exe --csharp_out="A:\Output directory\" CNTK.proto
Missing input file.
A:\proto_dir>protoc.exe --csharp_out="A:\" CNTK.proto
Missing input file.
A:\proto_dir>protoc.exe --csharp_out="A:/" CNTK.proto
A:\proto_dir>protoc.exe --csharp_out="A:/Output directory/" --proto_path=./ CNTK.proto
It might be easier to start with following the tutorials first: https://developers.google.com/protocol-buffers/docs/csharptutorial#compiling-your-protocol-buffers
@Ark-kun I second your criticism. This is just horrible. Instead of actual software devolopment I have to play this guessing games with command line :/
The cli really is badly designed. It could use more helpful messages when outputting errors.
I take particular issue with "File does not reside within any path specified using --proto_path (or -I). You must specify a --proto_path which encompasses this file. Note that the proto_path must be an exact prefix of the .proto file names -- protoc is too dumb to figure out when two paths (e.g. absolute and relative) are equivalent (it's harder than you think)"
Firstly, how do they know what I think. Secondly, it's not hard. Not even what I would remotely call a "hard" problem. Let's have a try, what's the normal steps to solve a problem like this.
- Google it. I tried googling "how to test if paths are equivalent C++ filesystem" the top result was this.
- Oh wait, step 1 solved it. Actually it was easier than I thought.
I have yet to find a way to toggle the verbosity of protoc. Is this possible? That would really help to double-check that the CLI arguments were set correctly.
@dfontenot There isn't a verbosity mode in protoc. Inside Google we rarely need to invoke protoc directly (the internal version of bazel takes care of everything) so not much attention has been given to the command line interface. If you have suggestions on how to improve the current error messages, feel free to send us pull requests.
I went through the list in https://github.com/google/protobuf/issues/3028#issue-225299414 and believe many of the commands should have worked. Created https://github.com/google/protobuf/issues/4993 to fix protoc on windows.
@xfxyjwf With all due respect the documentation link you've provided does not provide much help. Here is the single line mentioning protoc there:
protoc -I=$SRC_DIR --csharp_out=$DST_DIR $SRC_DIR/addressbook.proto
I have 17+ lines that conform to that documentation, yet fail to work:
protoc a:\proto_dir\CNTK.proto --csharp_out A:\Output directory\cntk.proto.cs --proto_path a:\proto_dir
protoc a:\proto_dir\CNTK.proto --csharp_out "A:\Output directory\cntk.proto.cs" --proto_path a:\proto_dir
protoc a:\proto_dir\CNTK.proto --csharp_out "A:\Output directory" --proto_path a:\proto_dir
protoc CNTK.proto --csharp_out "A:\Output directory" --proto_path a:\proto_dir
protoc a:\proto_dir\CNTK.proto --csharp_out "A:\Output directory" --proto_path a:\proto_dir protoc --csharp_out "A:\Output directory" --proto_path a:\proto_dir\ a:\proto_dir\CNTK.proto protoc --csharp_out "A:\Output directory" --proto_path a:\proto_dir\ CNTK.proto protoc --csharp_out "A:\Output directory\CNTK.proto.cs" a:\proto_dir\CNTK.proto protoc --csharp_out "A:\Output directory\CNTK.proto.cs" --proto_path=a:\proto_dir\ a:\proto_dir\CNTK.proto protoc --csharp_out "A:\Output directory" --proto_path=a:\proto_dir\ a:\proto_dir\CNTK.proto protoc --csharp_out="A:\Output directory" --proto_path="a:\proto_dir" a:\proto_dir\CNTK.proto protoc --csharp_out="A:\Output directory" --proto_path="a:\proto_dir" "a:\proto_dir\CNTK.proto" protoc --csharp_out="A:\Output directory" --proto_path="a:\proto_dir" "a:\proto_dir\CNTK.proto" A:\proto_dir>protoc.exe --csharp_out="A:\Output directory" --proto_path="a:\proto_dir" CNTK.proto A:\proto_dir>protoc.exe --csharp_out="A:\Output directory" --proto_path=. CNTK.proto A:\proto_dir>protoc.exe --csharp_out="A:\Output directory" --proto_path=./ CNTK.proto A:\proto_dir>protoc.exe --csharp_out="A:\Output directory" CNTK.proto
Hey @TeBoring I saw you added this to ToDO in "Weekly Fixit". I've gone and had a look at the path resolution code. There is a comment from "kenton" stating that in essence the current behaviour of disallowing ".." relative paths is to support some obscure setup where people have symbolic links in their source code. Symbolic links are there to hack around problems of inflexible software that hard codes paths. This is a perfect example of some hypothetical few being favoured over the many that would have much more hair if they were allowed to use relative paths to proto files. Please allow me to specify relative paths like normal software does.
It seems like no matter how you specify the path to the .proto file, protoc only works if the .proto file is lower in the folder hierarchy than where you called protoc. Your .proto file seems to need to be in a subfolder of the folder you call protoc in.
For example, you can never do this,
protoc --cpp_out=. ../myfile.proto
Even more strange, you can't even use an absolute path instead of ..:
protoc --cpp_out=. /path/to/myfile.proto
Using proto_path seems to have no effect.
However this will work:
protoc --cpp_out=. somefolder/myfile.proto
Sorry about all the frustration on this point. There is indeed some surprising behavior in protoc's command-line UX.
It seems like it would be useful if we created a short page that explains how to invoke protoc, along with highlighting the surprising cases (like how proto_path is used and how it defaults to the current working directory). We could then link to that doc from some of these error messages to hopefully help users resolve the issues more quickly.
Perhaps fixing the issue might have more impact. Sometimes it might even be easier to fix an issue than to document the problematic behavior.
Yeah, completely agree!
On Tue, 27 Sep 2022 at 3:27 am, Alexey Volkov @.***> wrote:
Perhaps fixing the issue might have more impact. Sometimes it might even be easier to fix an issue than to document the problematic behavior.
— Reply to this email directly, view it on GitHub https://github.com/protocolbuffers/protobuf/issues/3028#issuecomment-1258515914, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACQOQY4KP3TTHCU3WZNLK2DWAH2L3ANCNFSM4DJRZCRA . You are receiving this because you commented.Message ID: @.***>
No disagreement there Protoc, especially the pathing, is full of these weird, completely unexplained logics - the one line in the tutorial doesn't count as explanation. So instead of developing, I have to play this stupid game of "guess precisely how the CLI wants to be fed this path" For instance: If the tutorial had a paragraph saying something like "If your File structure looks like: Current Directory Subdirectory test.proto Subdirectory2 Then inputs would look like <Example input>". Many cumulative hours and much frustration would be saved.
We triage inactive PRs and issues in order to make it easier to find active work. If this issue should remain active or becomes active again, please add a comment.
This issue is labeled inactive because the last activity was over 90 days ago.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
We triage inactive PRs and issues in order to make it easier to find active work. If this issue should remain active or becomes active again, please reopen it.
This issue was closed and archived because there has been no new activity in the 14 days since the inactive label was added.