Incompatible with protobuf 30.0

[christian-heusel]

Hello hello, Arch Linux packager here 👋🏻

We have recently started the protobuf 30.0 rebuild and the hefur package is one of the holdups that came up along the way due to the following compiler error:

/build/hefur/src/hefur/mimosa/mimosa/rpc/gen/protoc-gen-mimosa.cc:7:10: fatal error: google/protobuf/compiler/cpp/generator.h: No such file or directory
    7 | #include <google/protobuf/compiler/cpp/generator.h>
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
make[2]: *** [mimosa/mimosa/rpc/gen/CMakeFiles/protoc-gen-mimosa.dir/build.make:79: mimosa/mimosa/rpc/gen/CMakeFiles/protoc-gen-mimosa.dir/protoc-gen-mimosa.cc.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:1772: mimosa/mimosa/rpc/gen/CMakeFiles/protoc-gen-mimosa.dir/all] Error 2

I think this is due to a few string interface changes, see for example the changes from https://github.com/protobuf-c/protobuf-c/pull/762 🤗

[abique]

Hi, I've had a first look on the macbook, don't pull yet, I'll finish tomorrow.

[abique]

@christian-heusel Hi, please can you try to compile the master branch with protoc 30? I've added some preprocessor to be able to target both versions.

[christian-heusel]

@dbermond could you look into this? 😊

[dbermond]

The git master branch (commit 90237b7b2fc6244c23855a5b0192dc60a3d3230f) still fails to compile with protobuf 30.0.

Same error:

/build/hefur/src/hefur/mimosa/mimosa/rpc/gen/protoc-gen-mimosa.cc:8:10: fatal error: google/protobuf/compiler/cpp/cpp_generator.h: No such file or directory
    8 | #include <google/protobuf/compiler/cpp/cpp_generator.h>
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
make[2]: *** [mimosa/mimosa/rpc/gen/CMakeFiles/protoc-gen-mimosa.dir/build.make:79: mimosa/mimosa/rpc/gen/CMakeFiles/protoc-gen-mimosa.dir/protoc-gen-mimosa.cc.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:1772: mimosa/mimosa/rpc/gen/CMakeFiles/protoc-gen-mimosa.dir/all] Error 2

Note: mimosa submodule is at correct checkout:

Submodule path 'mimosa': checked out 'cb52da44a9d1e30a3ddd167eaf7847796116eefa'

[abique]

So the error is different, this times it failed to include cpp_generator.h which shows that the preprocessor checks worked.

According to https://protobuf.dev/reference/cpp/api-docs/google.protobuf.compiler.cpp_generator/ I'm doing the correct include 🤔

Can I download your protobuf 30 package somewhere?

[dbermond]

Can I download your protobuf 30 package somewhere?

Sure, here. In this page there is also a list of files shipped in the package.

Direct download: link.

[christian-heusel]

I think this is due to this change https://github.com/protocolbuffers/protobuf/commit/b4b93b36245a88a3efa19261638f5da61a9be44f 🤔

[abique]

This looks like a breaking change from protobuf. I wonder if this is a mistake and if you should wait for 30.1 🤔

As I've pointed out in the documentation link, the compiler API is documented and doesn't seem to be a private implementation detail.

[christian-heusel]

I'll look into it a bit more and report to protobuf if neccessary 😅

[abique]

I'll look into it a bit more and report to protobuf if neccessary 😅

Thanks 👍

On my end, I don't think I'll replace protobuf by something else anytime soon and if I were to do some work on hefur, that'd be a rewrite in rust instead, but I don't think I have time for that in the next year.

[abique]

Though I have ideas, on how linux packages distributions could be done using bittorrent and a smarter tracker instead of mirrorring and letting the user choose a mirror for http download.

[christian-heusel]

@abique did you see the protobuf' upstream comment? 🤔 Their stance seems to be that these headers were always private 🤔

[abique]

@abique did you see the protobuf' upstream comment? 🤔 Their stance seems to be that these headers were always private 🤔

Then I learned something ;-) They didn't close this issue yet, so maybe they add the headers back. I believe it'd be a mistake to remove those headers now that they have been public and documented for so many years, and without a notice at all.

So this are the options:

1. change/patch protobuf so it ships again the "private" headers
2. manage to build hefur against protobuf 29, maybe statically, could be done using vcpkg
3. hefur is dead and gone as a consequence of protobuf 30
4. hefur is built without RPC and web interface. (protobuf is gone)

[eve-atum]

#3 would probably make a lot of people sad. I have no idea what the usage stats are like, but hefur was the only light-weight, stand-alone torrent tracker I could find while preparing for the 7th Heaven v2.0 release.

[dbermond]

5. add a protobuf git submodule ;)

[abique]

5. add a protobuf git submodule ;)

Yeah right, we could do that.

[dbermond]

When trying to use the headers from a local copy of protobuf sources (v30.0), the error changes to:

/build/hefur/src/hefur/mimosa/mimosa/rpc/gen/protoc-gen-mimosa.cc: In static member function ‘static uint32_t ServiceGenerator::computeMethodId(const google::protobuf::MethodDescriptor*)’:
/build/hefur/src/hefur/mimosa/mimosa/rpc/gen/protoc-gen-mimosa.cc:32:34: error: cannot convert ‘google::protobuf::internal::DescriptorStringView’ {aka ‘std::basic_string_view<char>’} to ‘const std::string&’ {aka ‘const std::__cxx11::basic_string<char>&’}
   32 |     return computeId(method->name());
      |                      ~~~~~~~~~~~~^~
      |                                  |
      |                                  google::protobuf::internal::DescriptorStringView {aka std::basic_string_view<char>}
/build/hefur/src/hefur/mimosa/mimosa/rpc/gen/protoc-gen-mimosa.cc:24:49: note:   initializing argument 1 of ‘static uint32_t ServiceGenerator::computeId(const std::string&)’
   24 |   static uint32_t computeId(const std::string & str)
      |                             ~~~~~~~~~~~~~~~~~~~~^~~

Many of these appear.

Is this something addressable?

[abique]

Though, I could probably rewrite a better hefur within 50 hours in rust. 🤔

[abique]

When trying to use the headers from a local copy of protobuf sources (v30.0), the error changes to:

/build/hefur/src/hefur/mimosa/mimosa/rpc/gen/protoc-gen-mimosa.cc: In static member function ‘static uint32_t ServiceGenerator::computeMethodId(const google::protobuf::MethodDescriptor*)’:
/build/hefur/src/hefur/mimosa/mimosa/rpc/gen/protoc-gen-mimosa.cc:32:34: error: cannot convert ‘google::protobuf::internal::DescriptorStringView’ {aka ‘std::basic_string_view<char>’} to ‘const std::string&’ {aka ‘const std::__cxx11::basic_string<char>&’}
   32 |     return computeId(method->name());
      |                      ~~~~~~~~~~~~^~
      |                                  |
      |                                  google::protobuf::internal::DescriptorStringView {aka std::basic_string_view<char>}
/build/hefur/src/hefur/mimosa/mimosa/rpc/gen/protoc-gen-mimosa.cc:24:49: note:   initializing argument 1 of ‘static uint32_t ServiceGenerator::computeId(const std::string&)’
   24 |   static uint32_t computeId(const std::string & str)
      |                             ~~~~~~~~~~~~~~~~~~~~^~~

Many of these appear.

Is this something addressable?

Yes it is. If you have an arch package with those headers, I'll grab it and fix the compilation issues.

[dbermond]

Currently, there is no Arch Linux package for protobuf 30.0 with these headers. You can use the headers from the protobuf 30.0 source tarball, then point CXXFLAGS to the headers location, which is the src directory. That's what I did, but using the Arch Linux package building tools.

This can be translated to something like:

$ curl -sO https://github.com/protocolbuffers/protobuf/releases/download/v30.0/protobuf-30.0.tar.gz
$ tar -x -f protobuf-30.0.tar.gz
$ export PROTOBUF_SRC="$(pwd)/protobuf-30.0"
$ export CXXFLAGS+=" -isystem${PROTOBUF_SRC}/src"
$ # <hefur build commands>

EDIT: fixed directories.

[christian-heusel]

Many of these appear.

Is this something addressable?

Yeah this is not too hard to do, you just need to sparkle in some conversations to std::string 😊 See for example https://github.com/astroidmail/astroid/pull/763 or https://github.com/USBGuard/usbguard/pull/650

[christian-heusel]

Did you already have a chance to look into this @dbermond? hefur is now the last package after I patched a few more on the list 😊

[dbermond]

Did you already have a chance to look into this @dbermond? hefur is now the last package after I patched a few more on the list 😊

Done! Thanks @christian-heusel and @abique for the assistance!

[abique]

Hi,

I've updated hefur and mimosa. Somehow your distribution of protobuf 30 doesn't include google/protobuf/compiler/cpp/cpp_generator.h so I can't compile hefur on my machine.

Cheers, Alex

[dbermond]

Somehow your distribution of protobuf 30 doesn't include google/protobuf/compiler/cpp/cpp_generator.h so I can't compile hefur on my machine.

Protobuf removed this header in version 30.0. Now you need to use the protobuf sources for providing the removed ones, as I already explained.

[abique]

OK, so at least I did the compilation fixes with the std::string errors.

[abique]

It seems we're getting the headers back! https://github.com/protocolbuffers/protobuf/commit/0b1a9c99c4238fc45cbc685dffc89ea8b69b6e5a