How to serialize google.protobuf.Timestamp in JSON?

[onx2]

trafficstars

I'm using Postman to make requests to my gRPC server and have some timestamps that are not being serialized in the JSON response.

I wondering if I'm leaving something out in my config or Cargo.toml, or maybe just misunderstood how this package works.

My Cargo.toml includes:

[dependencies]
tonic = "0.7.2"
prost = { version = "0.10.4", features = ["prost-derive"] }
prost-types = "0.10.1"
chrono = "0.4.19"
serde = "1.0.137"
serde_json = "1.0.81"

[build-dependencies]
tonic-build = "0.7.2"

Given a proto:

syntax = "proto3";
package user;

import "google/protobuf/timestamp.proto";

service UserService {
  rpc GetUser(GetUserRequest) returns (User);
}

message User {
  string id = 1;
  optional string email = 2;
  google.protobuf.Timestamp created_at = 3;
  google.protobuf.Timestamp updated_at = 4;
}

message GetUserRequest { string id = 1; }

I'd expect the response to be something like:

{
  "id": "fb0ab2ca-773f-40ac-a7c4-c32df39ee3dd",
  "created_at": ""2022-06-29T02:08:07.100Z"",
  "updated_at": ""2022-06-29T02:08:07.100Z"",
}

but instead I get the Timestamp response of something like (ignore actual dates, they wont match):

{
  "id": "fb0ab2ca-773f-40ac-a7c4-c32df39ee3dd",
  "created_at": {
    "seconds": "1656466246",
    "nanos": 85158000
  },
  "updated_at": {
    "seconds": "1656468195",
    "nanos": 841387000
  }
}

I've tried to add the type_attribute to the build like below but I get an error.

.type_attribute(
    ".",
    "#[derive(serde::Serialize,serde::Deserialize)]",
)

[LucioFranco]

You can do the type_attribute step but then also compile_well_known_types and it should codegen it for you.

[onx2]

That doesn't work for me unfortunately. I tried a few variations and created a new project with the bare minimum with no luck. However, I don't think I necessarily need Timestamp since string is okay enough.

Here's the output when I try to add Timestamp without that compile_well_known_types option:

error[E0277]: the trait bound `prost_types::Timestamp: Serialize` is not satisfied
    --> /home/onx2/Documents/test_proto/target/debug/build/test_proto-e07016b4dd081986/out/user.rs:12:5
     |
12   |     #[prost(message, optional, tag="5")]
     |     ^ the trait `Serialize` is not implemented for `prost_types::Timestamp`
     |
     = note: required because of the requirements on the impl of `Serialize` for `std::option::Option<prost_types::Timestamp>`
note: required by a bound in `models::_::_serde::ser::SerializeStruct::serialize_field`
    --> /home/onx2/.cargo/registry/src/github.com-1ecc6299db9ec823/serde-1.0.138/src/ser/mod.rs:1899:12
     |
1899 |         T: Serialize;
     |            ^^^^^^^^^ required by this bound in `models::_::_serde::ser::SerializeStruct::serialize_field`

build.rs

fn main() {
    let proto_files = ["./proto/user.proto"];

    tonic_build::configure()
        .build_client(false)
        .build_server(true)
        .type_attribute(".", "#[derive(serde::Serialize, serde::Deserialize)]")
        .compile(&proto_files, &["."])
        .unwrap_or_else(|e| panic!("protobuf compile error: {}", e));

    println!("cargo:rerun-if-changed={:?}", proto_files);
}

and with the option turned on:

pub created_at: ::core::option::Option<super::google::protobuf::Timestamp>,
   |                                               ^^^^^^ could not find `google` in `super`
error[E0698]: type inside `async` block must be known in this context
   --> /home/onx2/Documents/test_proto/target/debug/build/test_proto-e07016b4dd081986/out/user.rs:171:58
    |
171 |                         let res = grpc.unary(method, req).await;
    |                                                          ^^^^^^ cannot infer type
error[E0698]: type inside `async` block must be known in this context
   --> /home/onx2/Documents/test_proto/target/debug/build/test_proto-e07016b4dd081986/out/user.rs:171:58
    |
171 |                         let res = grpc.unary(method, req).await;
    |                                                          ^^^^^^ cannot infer type
    |
note: the type is part of the `async` block because of this `await`
   --> /home/onx2/Documents/test_proto/target/debug/build/test_proto-e07016b4dd081986/out/user.rs:171:58
    |
171 |                         let res = grpc.unary(method, req).await;
    |                                                          ^^^^^^

error[E0283]: type annotations needed
 --> /home/onx2/Documents/test_proto/target/debug/build/test_proto-e07016b4dd081986/out/user.rs:2:28
  |
2 | #[derive(Clone, PartialEq, ::prost::Message)]
  |                            ^^^^^^^^^^^^^^^^ cannot infer type
  |
  = note: cannot satisfy `_: Default`
  = note: this error originates in the derive macro `::prost::Message` (in Nightly builds, run with -Z macro-backtrace for more info)

and for context, here is what my User struct definition looks like. user.rs

use chrono::{DateTime, Utc};
use serde::{Deserialize, Serialize};
use uuid::Uuid;

#[derive(Debug, Serialize, Deserialize)]
pub struct User {
    pub id: Uuid,
    pub status: i32,
    pub created_at: DateTime<Utc>,
    pub updated_at: Option<DateTime<Utc>>,
}

[onx2]

The first compiler error makes me think I'm missing a dependency or config option since it's saying Serialize isn't implemented for prost timestamp.

Any suggestions?

[LucioFranco]

You will need to use compile_well_known_types for this to work since without that option prost pulls in prost-types for these well known types and it doesn't have serde support.

[onx2]

I will double check in a minimal replication repo but that option caused other issues for me and my project wouldn't compile as expected. I'll post results later.

[LucioFranco]

If you have a minimal reproduction of the issues I can take a look at why that doesn't work.

[liufuyang]

I think I encountered some very similar issues here.

You can checkout this branch tmp-for-fixing-tonic-build-issue https://github.com/liufuyang/bigtable_rs/tree/tmp-for-fixing-tonic-build-issue

And do a git submodule init

Then you can try cargo build.

Similar errors as posted above will show up. And one can see different build errors by switching that compile_well_known_types on and off

Error with compile_well_known_types as off

error[E0277]: the trait bound `prost_types::Duration: Serialize` is not satisfied
    --> bigtable_rs/src/google/google.bigtable.v2.rs:712:5
     |
712  |     /// The latency measured by the frontend server handling this request, from
     |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ the trait `Serialize` is not implemented for `prost_types::Duration`
     |
     = help: the following other types implement trait `Serialize`:
               &'a T
               &'a mut T
               ()
               (T0, T1)
               (T0, T1, T2)
               (T0, T1, T2, T3)
               (T0, T1, T2, T3, T4)
               (T0, T1, T2, T3, T4, T5)
             and 213 others
     = note: required because of the requirements on the impl of `Serialize` for `std::option::Option<prost_types::Duration>`

Error with compile_well_known_types as on

   Compiling bigtable_rs v0.1.6 (/Users/fuyangl/workspace/bigtable_rs/bigtable_rs)
error[E0433]: failed to resolve: could not find `protobuf` in `super`
   --> bigtable_rs/src/google/google.bigtable.v2.rs:723:71
    |
723 |     pub frontend_server_latency: ::core::option::Option<super::super::protobuf::Duration>,
    |                                                                       ^^^^^^^^ could not find `protobuf` in `super`

error[E0433]: failed to resolve: could not find `protobuf` in `super`
   --> bigtable_rs/src/google/google.bigtable.v2.rs:931:70
    |
931 |         pub family_name: ::core::option::Option<super::super::super::protobuf::StringValue>,
    |                                                                      ^^^^^^^^ could not find `protobuf` in `super`

error[E0433]: failed to resolve: could not find `protobuf` in `super`
   --> bigtable_rs/src/google/google.bigtable.v2.rs:940:68
    |
940 |         pub qualifier: ::core::option::Option<super::super::super::protobuf::BytesValue>,
    |                                                                    ^^^^^^^^ could not find `protobuf` in `super`

error[E0283]: type annotations needed
   --> bigtable_rs/src/google/google.bigtable.v2.rs:710:28
    |
710 | #[derive(Clone, PartialEq, ::prost::Message)]
    |                            ^^^^^^^^^^^^^^^^ cannot infer type
    |
    = note: cannot satisfy `_: Default`
    = note: this error originates in the derive macro `::prost::Message` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0283]: type annotations needed
   --> bigtable_rs/src/google/google.bigtable.v2.rs:914:32
    |
914 |     #[derive(Clone, PartialEq, ::prost::Message)]
    |                                ^^^^^^^^^^^^^^^^ cannot infer type
    |
    = note: cannot satisfy `_: Default`
    = note: this error originates in the derive macro `::prost::Message` (in Nightly builds, run with -Z macro-backtrace for more info)

[LucioFranco]

With compile_well_known on it seems like you're not setting up the includes correctly. Are you using the include_file config option?

[horacimacias]

@liufuyang I had a similar super::super issue when using compile_well_known_types. I checked your bigtable_rs and I think you have several issues:

1. when you use compile_well_known_types you're basically not getting several extern_path elements introduced by prost, https://github.com/tokio-rs/prost/blob/cdbd148eea999a5e106260483bc12ef10686f0f7/prost-build/src/extern_paths.rs#L35-L62 Using compile_well_known_types sets prost_types to false so you're not getting what's inside that if block https://github.com/tokio-rs/prost/blob/cdbd148eea999a5e106260483bc12ef10686f0f7/prost-build/src/lib.rs#L478-L481

You need to "redo" this extern_path elements that you don't have now but you need (in your case StringValue and ByteValue):

.extern_path(
            ".google.protobuf.BytesValue",
            "::prost::alloc::vec::Vec<u8>",
        )
        .extern_path(
            ".google.protobuf.StringValue",
            "::prost::alloc::string::String",
        )

(you can copy-paste most of the extern_path elements from prost in case you ever use more types that you want to handle externally)

2. some of the protos you're using is using Duration and you're also missing the extern_path for that:

.extern_path(".google.protobuf.Duration", "::prost_wkt_types::Duration")

Prost also sets a 'blanket' extern_paths.insert(".google.protobuf", "::prost_types"); which in your case would be extern_paths.insert(".google.protobuf", "::prost_wkt_types"); if you want to use prost_wkt_types. This contains the Duration as well so you can choose whether to use this or to add the previous extern_path with only Duration.

With these changes I was able to get cargo build working on your project.

[liufuyang]

@horacimacias Oh I see, thank you so much for the help, this for sure makes my life easier.

     .extern_path(
         ".google.protobuf.BytesValue",
         "::prost::alloc::vec::Vec<u8>",
     )
     .extern_path(
         ".google.protobuf.StringValue",
         "::prost::alloc::string::String",
     )
     .extern_path(".google.protobuf", "::prost_wkt_types")

So in the end I added those as you suggested, it works now. Appreciated for the input 👍

[horacimacias]

@onx2 similar thing for you, I believe this error with the option turned on:

pub created_at: ::core::option::Option<super::google::protobuf::Timestamp>,
   |                                               ^^^^^^ could not find `google` in `super`

is caused by a missing

extern_paths.insert(".google.protobuf", "::prost_types")

or something close to that. If, in addition, you need to serialize/deserialize Timestamp, prost-wkt-types may help so something like

extern_paths.insert(".google.protobuf", "::prost-wkt-types")

may help (and add prost-wkt-types to your Cargo.toml)