Implement reflection-based automatic JSON parsing/serialization

[itrofimow]

Userver actively uses boost::pfr, which allows one to access an aggregate field by index, get total number of fields in an aggregate and a lot of other very handy things (grep pfr::for_each_field across the source tree).

This works wonders when one needs to map an aggregate to some kind of its binary representation, say, in database protocols (ClickHouse/Pg/MySQL), where there is a fixed format with fixed amount of fields, mapped 1 to 1. However, there was no way to get pfr working with JSON without adding some additional metadata: pfr, being a thing of beauty, unfortunately didn't have a way to get a field name, which is needed for textual formats (JSON/Yaml/etc.).

Well, this is no longer the case: with PRF 2.2.0 being just landed in userver, it IS possible to get a field name of an aggregate at compile time (with C++20): https://www.boost.org/doc/libs/master/doc/html/boost_pfr/tutorial.html#boost_pfr.tutorial.reflection_of_field_name, and the JSON/Yaml/Bson/etc. limitation of applying pfr to is lifted now.

---

It would be nice to have formats::Parse/formats::Serialize analogues built on top of this functionality, and from a distance i'd suggest implementing it in this order:

1. Add the functionality in question to an aggregate consisting of primitive types: std::string, integer types, floating-point types, bool:

struct Data { 
    std::string name; 
    int value; 
    bool is_something; 
    double amount;
};

2. Expand the functionality to work with std::optional<primitive type> (and std::variant if you are feeling adventurous)
3. Expand the functionality to work with nested aggregates (this adds JSON object, basically)
4. Expand the functionality to work with arrays of all the above (this adds JSON array)

Implementing this for std::chrono:: types could be quite handy as well, but that might potentially require some policies added. Would be nice to also have a room to hook into the auto-generated parsing for some types that aren't recognized out of the box (say, if one wants to have everything automatic but for one field, which requires some custom handling).

I would also suggest to share your design ideas before jumping on the implementation: this could potentially save a lot of time and prevent discovering design-level problems after everything is already implemented.

P.S. All of this should be ifdef-ed under C++20 availability, of course, and https://github.com/userver-framework/userver/blob/86b1c6c0307be0c0ce76b009a9db90b8667a4bd2/cmake/SetupEnvironment.cmake#L19 could be used for testing.

---

This is a really nice, fun and very useful feature, which would not only allow one to improve template metaprogramming and design skills, but also earn a lot of appreciation from the userver-community, so don't hesitate to give it a try, otherwise we may implement it ourselves at some point :wink:

[sabudilovskiy]

I know that there is an implementation of traversing structures with names in boost pfr, but in my opinion it is not very good to lay on it because: firstly it is tied to the intrinsics of compilers and can potentially break at any moment secondly it does a lot of instantiations on one structure, it seems to me that it will potentially slow down a lot in a large project. What's the alternative? I have an implementation on NTTP, traits, templates, constexp function and lite macros(declare enum). At the moment I have implemented basic arrays and restrictions on them, string and restrictions on them, structures, as well as the auto parse/serialize of handlers, auto-generation of openapi description from all this, wrote the necessary things for the postgres driver. Although my implementation is made solely out of interest and my own needs, but it seems to me that it can be considered. Formally, everything is covered by tests, it is really possible to approach this more thoroughly. Example array with restriction and tests: https://github.com/sabudilovskiy/timetable_vsu_backend/blob/new_openapi/utests/openapi/json/parse/basic_array.cpp Example of view: Type in body: https://github.com/sabudilovskiy/timetable_vsu_backend/blob/new_openapi/src/models/lesson_filter/type.hpp Declaration request and response: https://github.com/sabudilovskiy/timetable_vsu_backend/blob/new_openapi/src/views/timetable/declarations.hpp View: https://github.com/sabudilovskiy/timetable_vsu_backend/blob/new_openapi/src/views/timetable/view.cpp The openapi scheme is generated at the start of the service based on the information from the config (since the endpoint type and its path are set there), and then it is given by /openapi. There is no difficulty for me to change this behavior. This behavior is tested here: https://github.com/sabudilovskiy/timetable_vsu_backend/blob/new_openapi/utests/openapi/doc/serialize/basic_path.cpp I also want to add that now all the traits are implemented through types with static fields, but this caused problems if necessary to update them. I have a local version where all the traits have become Nttp, and the constexp strings have become fixed-size so that it is easier to work with them. How do you look at such an implementation? I would be interested to at least hear an opinion. P.s.Here I will leave different links to implementations of different pieces.

1. implementation of abstract handler https://github.com/sabudilovskiy/timetable_vsu_backend/blob/new_openapi/src/openapi/http/handler.hpp
2. test with body and header in response: https://github.com/sabudilovskiy/timetable_vsu_backend/blob/new_openapi/utests/openapi/http/basic_http_response.cpp
3. test with body, cookie and header in request: https://github.com/sabudilovskiy/timetable_vsu_backend/blob/new_openapi/utests/openapi/http/basic_http_request.cpp
4. full complete impl enums: macro to declare: https://github.com/sabudilovskiy/timetable_vsu_backend/blob/new_openapi/src/openapi/enum/declare.hpp enumerator func to pg: https://github.com/sabudilovskiy/timetable_vsu_backend/blob/new_openapi/src/openapi/enum/enumerator_func.hpp enum to OpenApi Doc: https://github.com/sabudilovskiy/timetable_vsu_backend/blob/new_openapi/src/openapi/doc/enum.hpp
5. generate OpenApi doc in this folder: https://github.com/sabudilovskiy/timetable_vsu_backend/tree/new_openapi/src/openapi/doc For example, structures: https://github.com/sabudilovskiy/timetable_vsu_backend/blob/new_openapi/src/openapi/doc/reflective.hpp
6. My structures can use additionalProperties: true, examples: serialize: https://github.com/sabudilovskiy/timetable_vsu_backend/blob/new_openapi/utests/openapi/json/serialize/basic_object.cpp#L36 parse: https://github.com/sabudilovskiy/timetable_vsu_backend/blob/new_openapi/utests/openapi/json/parse/basic_object.cpp#L59 openapi: https://github.com/sabudilovskiy/timetable_vsu_backend/blob/new_openapi/src/openapi/doc/reflective.hpp
7. example of generated openapi schema by my server: https://pastebin.com/9vpFZgES This openapi generated from all OpenApiViews and settings in static config: https://github.com/sabudilovskiy/timetable_vsu_backend/blob/new_openapi/configs/static_config.yaml.in#L101
8. impl openapi::types::Array, it can use min, max and uniqueItems as restrictions and user can write it in any order: https://github.com/sabudilovskiy/timetable_vsu_backend/blob/new_openapi/src/openapi/types/array.hpp

[apolukhin]

@sabudilovskiy your approach is very interesting, but it is a totally different approach that requires explicit markup in code. We'll take a closer look, which approach suits our needs better

[sabudilovskiy]

@sabudilovskiy your approach is very interesting, but it is a totally different approach that requires explicit markup in code. We'll take a closer look, which approach suits our needs better

Initially, this was done based on the fact that there was no other way to provide names for the fields. However, now everything is different. I think it is possible to remove the markup in most cases, however restrictions on openapi, an explicit indication of where the field will be placed in the request (header/body/cookie) would be easiest to do this way. In the near future I want to use boost.pfr.name and standard types so that I don't have to markup anything in most cases (however markup is required in requests and responses). Or is there some other way for requests and responses?

[linuxnyasha]

https://github.com/linuxnyasha/Serialization-userver_formats I wrote this. Could this be added to userver in the future? There's not much here yet, but it can all be added.