odygrd
feature request: support `quill_(format|encode)_as`
Is your feature request related to a problem? Please describe. I use this library extensively and I wonder if there's some more convenient way to log UDT with this library.
For a relatively large/not trivially copyable UDT, typically we only need to log several essential fields.
eg.
struct SuperBigStruct {
std::map<std::string, int> superBigMap;
}
template<>
struct fmt::formatter<SuperBigStruct> {
template <typename FCtx>
auto format(const SuperBigStruct &obj, FCtx &ctx) const {
return fmt::format(ctx.out(), "SuperBigStruct(size={})", obj.superBigMap.size());
}
};
So logically if we want to log this UDT we only need to encode & decode one size_t (which is obj.superBigMap.size()), but this is impossible under the current design structure(which always requires you to encode and decode the entire struct).
It is possible to do this though:
struct SuperBigStruct {
std::map<std::string, int> superBigMap;
}
struct SuperBigStructFormatProxy {
int size;
};
auto format_as(const SuperBigStruct &obj) { return SuperBigStructFormatProxy{obj.size()}; }
template<>
struct fmt::formatter<SuperBigStructFormatProxy> {
template <typename FCtx>
auto format(const SuperBigStructFormatProxy&obj, FCtx &ctx) const {
return fmt::format(ctx.out(), "SuperBigStruct(size={})", obj.size);
}
};
but this is too verbose.
Describe the solution you'd like
I think a way to automatically extract necessary fields from the format string to do encode/decode would be very nice.
I was browsing through the documentation of fmt the other day and I came across this std::make_format_args. Maybe we can support codec for the returned fmt::basic_format_args (which is essentially a list of references to the arguments).
So maybe something like this(but I don't know how to exactly implement it yet):
struct SuperBigStruct {
std::map<std::string, int> superBigMap;
int size;
// here we define the format string and pack of args for the format string
auto GetFormatArgs() const { return std::make_format_args(size); }
static inline constexpr std::string_view kFmtStr{"SuperBigStruct(size={})"};
}
using ArgsType = std::invoke_result_t<&SuperBigStruct::GetFormatArgs, SuperBigStruct>;
auto format_as(const SuperBigStruct &obj) { return obj.GetFormatArgs(); }
// implement codec for the pack of args
// we get the list of args and encode those basic members one by one
// since basic_format_args is list of references, maybe we need to allocate & make the list of args point to
// the decoded value in backend thread.
template<>
quill::Codec<SuperBigStruct> : quill::FormatArgsCodec<ArgsType> {};
// formatter for ArgsType
template<>
fmtquill::formatter<ArgsType> {
template <typename FCtx>
auto format(const ArgsType &args, FCtx &ctx) const {
return fmt::vformat(ctx.out(), "SuperBigStruct(size={})", args);
}
}
Additional context
This is just my immature two cents, I'm open to discussion.
hey, regarding logging user-defined types with only selective members — there isn’t currently a less verbose way built-in. It is not a very common use case
-
as you correctly noted, you can do it via a proxy object. the downside is that you need to explicitly construct the proxy every time you log, e.g.
LOG_INFO("{}", SuperBigStructFormatProxy{sbs}); -
there’s another approach which requires more boilerplate up front, but afterwards you can just log like this:
LOG_INFO(logger, "{}", sbs);(i.e., no need to wrap the object manually each time)
see the example below
#include "quill/Backend.h"
#include "quill/Frontend.h"
#include "quill/LogMacros.h"
#include "quill/Logger.h"
#include "quill/sinks/ConsoleSink.h"
#include <iostream>
#include <string>
#include <utility>
#include <map>
struct SuperBigStruct {
std::map<std::string, int> superBigMap;
};
struct SuperBigStructProxy {
size_t size{};
};
/***/
template <>
struct fmtquill::formatter<SuperBigStructProxy>
{
constexpr auto parse(format_parse_context& ctx) { return ctx.begin(); }
auto format(::SuperBigStructProxy const& sbs, format_context& ctx) const
{
return fmtquill::format_to(ctx.out(), "size: {}", sbs.size);
}
};
template <>
struct quill::Codec<::SuperBigStruct>
{
static size_t compute_encoded_size(detail::SizeCacheVector& conditional_arg_size_cache, ::SuperBigStruct const& sbs) noexcept
{
return compute_total_encoded_size(conditional_arg_size_cache, sbs.superBigMap.size());
}
static void encode(std::byte*& buffer, detail::SizeCacheVector const& conditional_arg_size_cache,
uint32_t& conditional_arg_size_cache_index, ::SuperBigStruct const& sbs) noexcept
{
// You must encode the same members and in the same order as in compute_total_encoded_size
encode_members(buffer, conditional_arg_size_cache, conditional_arg_size_cache_index, sbs.superBigMap.size());
}
static ::SuperBigStructProxy decode_arg(std::byte*& buffer)
{
// You must decode the same members and in the same order as in encode
::SuperBigStructProxy sbs;
decode_members(buffer, sbs, sbs.size);
return sbs;
}
static void decode_and_store_arg(std::byte*& buffer, DynamicFormatArgStore* args_store)
{
args_store->push_back(decode_arg(buffer));
}
};
int main()
{
quill::BackendOptions backend_options;
quill::Backend::start(backend_options);
// Frontend
auto console_sink = quill::Frontend::create_or_get_sink<quill::ConsoleSink>("sink_id_1");
quill::Logger* logger = quill::Frontend::create_or_get_logger("root", std::move(console_sink));
SuperBigStruct sbs1;
sbs1.superBigMap.insert({"key1", 1});
SuperBigStruct sbs2;
sbs2.superBigMap.insert({"key1", 1});
sbs2.superBigMap.insert({"key2", 2});
LOG_INFO(logger, "{} {}", sbs1, sbs2);
}
It is not a bad idea to have this type of Codec in the library, my main concern is that we’d have to standardize on a fixed function name, like quill_format_as(), and require users to implement that inside their UDTs.
See the below implementation which you can add locally for now, I think it does what you need.
Note 1: it calls sbs.quill_format_as() twice on the hot path.
Note 2: I haven't tested it extensively but it seems to be working
#include "quill/Backend.h"
#include "quill/Frontend.h"
#include "quill/LogMacros.h"
#include "quill/Logger.h"
#include "quill/sinks/ConsoleSink.h"
#include <iostream>
#include <map>
#include <string>
#include <utility>
#include "quill/bundled/fmt/format.h"
#include <tuple>
template <typename T>
struct FormatAsTupleCodec
{
static size_t compute_encoded_size(quill::detail::SizeCacheVector& conditional_arg_size_cache, T const& sbs) noexcept
{
size_t result = 0;
std::apply(
[&conditional_arg_size_cache, &result](auto&&... args)
{
// Fold expression to sum up the encoded size of each argument
result = (result + ... +
quill::Codec<std::decay_t<decltype(args)>>::compute_encoded_size(
conditional_arg_size_cache, args));
},
sbs.quill_format_as());
return result;
}
static void encode(std::byte*& buffer, quill::detail::SizeCacheVector const& conditional_arg_size_cache,
uint32_t& conditional_arg_size_cache_index, T const& sbs) noexcept
{
std::apply(
[&buffer, &conditional_arg_size_cache, &conditional_arg_size_cache_index](auto&&... args)
{
(quill::Codec<quill::detail::remove_cvref_t<decltype(args)>>::encode(
buffer, conditional_arg_size_cache, conditional_arg_size_cache_index, args),
...);
},
sbs.quill_format_as());
}
static std::string decode_arg(std::byte*& buffer)
{
// Get the type of the tuple returned by quill_format_as()
using format_tuple_type = decltype(std::declval<T>().quill_format_as());
format_tuple_type decoded_tuple;
decode_tuple_elements(buffer, decoded_tuple,
std::make_index_sequence<std::tuple_size_v<format_tuple_type>>{});
return format_from_tuple(decoded_tuple);
}
// Helper to decode each tuple element individually
template <typename Tuple, size_t... Is>
static void decode_tuple_elements(std::byte*& buffer, Tuple& tuple, std::index_sequence<Is...>)
{
((std::get<Is>(tuple) = quill::Codec<std::decay_t<std::tuple_element_t<Is, Tuple>>>::decode_arg(buffer)), ...);
}
// Helper to apply formatting with the tuple elements
template <typename Tuple>
static std::string format_from_tuple(Tuple const& tuple)
{
return std::apply([](auto const& format_str, auto const&... args)
{ return fmtquill::format(format_str, args...); }, tuple);
}
static void decode_and_store_arg(std::byte*& buffer, quill::DynamicFormatArgStore* args_store)
{
args_store->push_back(decode_arg(buffer));
}
};
/**
* Example
*/
struct SuperBigStruct
{
std::map<std::string, int> superBigMap;
auto quill_format_as() const noexcept
{
return std::make_tuple(std::string{"size: {}"}, superBigMap.size());
}
};
template <>
struct quill::Codec<SuperBigStruct> : FormatAsTupleCodec<SuperBigStruct>
{
};
int main()
{
quill::BackendOptions backend_options;
quill::Backend::start(backend_options);
// Frontend
auto console_sink = quill::Frontend::create_or_get_sink<quill::ConsoleSink>("sink_id_1");
quill::Logger* logger = quill::Frontend::create_or_get_logger("root", std::move(console_sink));
SuperBigStruct sbs1;
sbs1.superBigMap.insert({"key1", 1});
SuperBigStruct sbs2;
sbs2.superBigMap.insert({"key1", 1});
sbs2.superBigMap.insert({"key2", 2});
LOG_INFO(logger, "{} {}", sbs1, sbs2);
}
I think we can have something like the following as a solution
#include "quill/Backend.h"
#include "quill/Frontend.h"
#include "quill/LogMacros.h"
#include "quill/Logger.h"
#include "quill/sinks/ConsoleSink.h"
#include <map>
#include <string>
#include <utility>
#include "quill/bundled/fmt/format.h"
#include <tuple>
namespace quill
{
/**
* @brief Provides selective field serialization for large or complex user-defined types.
*
* This codec enables logging only specific fields of large objects instead of the entire structure,
* taking a smaller copy of the structure on the hot path.
*
* The codec serializes only the specified fields rather than the entire object,
* making it ideal for objects with many fields where only a few are relevant for logging.
*
* The user specifies which fields to include by
* implementing a static `format` method that returns a tuple containing:
* 1. A format string with placeholders
* 2. The selected field values to be formatted
*
* Usage example:
* @code
* template <>
* struct quill::Codec<VeryLargeClass> : SelectiveLogCodec<VeryLargeClass>
* {
* static auto format(VeryLargeClass const& obj)
* {
* return std::make_tuple("VeryLargeClass(id={}, count={})", obj.id, obj.count);
* }
* };
* @endcode
*/
template <typename T>
struct SelectiveLogCodec
{
static size_t compute_encoded_size(quill::detail::SizeCacheVector& conditional_arg_size_cache, T const& sbs) noexcept
{
size_t result = 0;
std::apply(
[&conditional_arg_size_cache, &result](auto&&... args)
{
// Fold expression to sum up the encoded size of each argument
result = (result + ... +
quill::Codec<std::decay_t<decltype(args)>>::compute_encoded_size(
conditional_arg_size_cache, args));
},
Codec<T>::format(sbs));
return result;
}
static void encode(std::byte*& buffer, quill::detail::SizeCacheVector const& conditional_arg_size_cache,
uint32_t& conditional_arg_size_cache_index, T const& sbs) noexcept
{
std::apply(
[&buffer, &conditional_arg_size_cache, &conditional_arg_size_cache_index](auto&&... args)
{
(quill::Codec<quill::detail::remove_cvref_t<decltype(args)>>::encode(
buffer, conditional_arg_size_cache, conditional_arg_size_cache_index, args),
...);
},
Codec<T>::format(sbs));
}
static std::string decode_arg(std::byte*& buffer)
{
// Get the type of the tuple returned by quill_format_as()
using format_tuple_type = decltype(Codec<T>::format(std::declval<T>()));
format_tuple_type decoded_tuple;
decode_tuple_elements(buffer, decoded_tuple,
std::make_index_sequence<std::tuple_size_v<format_tuple_type>>{});
return format_from_tuple(decoded_tuple);
}
// Helper to decode each tuple element individually
template <typename Tuple, size_t... Is>
static void decode_tuple_elements(std::byte*& buffer, Tuple& tuple, std::index_sequence<Is...>)
{
((std::get<Is>(tuple) = quill::Codec<std::decay_t<std::tuple_element_t<Is, Tuple>>>::decode_arg(buffer)), ...);
}
// Helper to apply formatting with the tuple elements
template <typename Tuple>
static std::string format_from_tuple(Tuple const& tuple)
{
return std::apply([](auto const& format_str, auto const&... args)
{ return fmtquill::format(format_str, args...); }, tuple);
}
static void decode_and_store_arg(std::byte*& buffer, quill::DynamicFormatArgStore* args_store)
{
args_store->push_back(decode_arg(buffer));
}
};
} // namespace quill
/**
* Example
*/
struct SuperBigStruct
{
explicit SuperBigStruct(int in_x) : x(in_x) {};
std::map<std::string, int> superBigMap;
int x;
};
template <>
struct quill::Codec<SuperBigStruct> : SelectiveLogCodec<SuperBigStruct>
{
static auto format(SuperBigStruct const& s)
{
return std::make_tuple("SuperBigStruct(size={}, x={})", s.superBigMap.size(), s.x);
}
};
int main()
{
quill::BackendOptions backend_options;
quill::Backend::start(backend_options);
// Frontend
auto console_sink = quill::Frontend::create_or_get_sink<quill::ConsoleSink>("sink_id_1");
quill::Logger* logger = quill::Frontend::create_or_get_logger("root", std::move(console_sink));
SuperBigStruct sbs1{100};
sbs1.superBigMap.insert({"key1", 1});
SuperBigStruct sbs2{200};
sbs2.superBigMap.insert({"key1", 1});
sbs2.superBigMap.insert({"key2", 2});
LOG_INFO(logger, "{} {}", sbs1, sbs2);
}
the downside is that you need to explicitly construct the proxy every time you log, e.g. LOG_INFO("{}", SuperBigStructFormatProxy{sbs});
I don't think so since I've specified format_as. When the object is being logged fmt will search for format_as in the same namespace and automatically log the struct of the converted type. https://github.com/fmtlib/fmt/issues/3266
template <typename T> struct SelectiveLogCodec
I think this is super nice but the downside is we have to specify quill::Codec::format which might be verbose and loses compatibility with fmt(unless you write the same code twice).
Maybe we can construct some struct that's the composition of format string and tuple of arguments and format_as to enforce fmt library to encode that, eg.
/* quill provides the following */
template<typename... Args>
struct FormatProxyType {
std::string_view fmtString;
std::tuple<Args...> args;
FormatProxyType(std::string_view fmtString, Args &&...args): fmtString(fmtString), args(std::move(args)...) {}
};
template<typename... Args>
struct fmtquill::formatter<FormatProxyType> {
template <typename FCtx>
auto format(const SuperBigStructFormatProxy&obj, FCtx &ctx) const {
// this does not actually work, need to use some trick with fmt::make_format_args
return fmt::format(ctx.out(), obj.fmtString, std::apply(fmt::make_format_args, obj.args));
}
};
// implement quill codec for formatproxytype(which I guess should be trivial)
/* user provides the following */
struct SuperBigStruct {
std::map<std::string, int> superBigMap;
}
auto format_as(const SuperBigStruct &obj) { return FormatProxyType("SuperBigStruct{}", obj.size); }
But this still pays the price of copying the arguments to std::tuple on the hot path and I don't think we can avoid that as long as we're using std::tuple directly. This could be even worse if we're logging some non-trivially copyable fields.
I think we can avoid this by using std::tuple with string_views and reference_wrappers:
struct SuperBigStruct {
std::string veryLongString;
std::queue<std::string> queueWithOneElement;
}
auto format_as(const SuperBigStruct &obj) {
return FormatProxyType("SuperBigStruct(header={:.8}, front={:.8})",
std::string_view(obj.veryLongString), std::reference_wrapper<...>(obj.queueWithOneElement)); // sorry for being lazy
}
I think this would work with fmt, but I don't think it will work with quill(does quill do remove_cvref on reference wrappers?).
Also this requires user to explicitly specify usage of std::reference_wrappers.
This is exactly the reason why I suggest implementing codec for fmt::base_format_args , because it is a list of references, not the concrete objects.
/* quill provides the following */
template<typename... Args>
struct FormatProxyType {
std::string_view fmtString;
fmt::base_format_args<Args...> args;
FormatProxyType(std::string_view fmtString, Args &&...args): fmtString(fmtString), args(fmt::make_format_args(std::move(args)...)) {}
};
template<typename... Args>
struct fmtquill::formatter<FormatProxyType> {
template <typename FCtx>
auto format(const SuperBigStructFormatProxy&obj, FCtx &ctx) const {
// this does not actually work, need to use some trick with fmt::make_format_args
return fmt::format(ctx.out(), obj.fmtString, std::apply(fmt::make_format_args, obj.args));
}
};
// implement quill codec for formatproxytype(which I guess is less trivial)
/* user provides the following */
struct SuperBigStruct {
std::string veryLongString;
std::queue<std::string> queueWithOneElement;
}
auto format_as(const SuperBigStruct &obj) {
return FormatProxyType("SuperBigStruct(header={:.8}, front={:.8})", obj.veryLongString, obj.queueWithOneElement);
}
Some time ago when I explored fmt::basic_format_args (the base_format_args), there wasn’t a public API to iterate over it or extract each argument’s type. This is crucial because we need to iterate arguments both in compute_size and encode. Then, on decode, we’d need to reconstruct base_format_args accordingly.
So, I’m not entirely sure how straightforward it would be to directly use base_format_args from fmt for this purpose.
But you get the general idea: with the tuple codec approach, it might be possible to give base_format_args a try as the underlying mechanism.
Regarding the previous solution for references, we could support something like this:
std::make_tuple("SuperBigStruct(size={}, x={})", s.superBigMap.size(), std::ref(s.x));
As for interoperability with fmt, yes, you’d need to provide a separate specialization for fmt::formatter for your user defined type. That said, there might be different use cases: for example, you might want to output everything when formatting for fmt, but selectively pass only some fields to the logger for efficiency or privacy reasons.
the downside is that you need to explicitly construct the proxy every time you log, e.g. LOG_INFO("{}", SuperBigStructFormatProxy{sbs});
I don't think so since I've specified
format_as. When the object is being loggedfmtwill search forformat_asin the same namespace and automatically log the struct of the converted type. fmtlib/fmt#3266
My mistake. format_as won't be respected in the hot path.
That being considered, I think this solution worths some further discussion:
quill_format_as()
- I don't think it needs to be implemented as a (static) member function, it can be implemented as a regular function under the same namespace with the struct that needs to be encoded, like the way
fmtsolves this withformat_as. - I think
quill_encode_asmight be a better name.
In this way user can specify the way to encode their UDTs to STL structs with minimum boilerplate code. If we could add codec support for FormatProxyType and let user use it in quill_format_as it will be even better.