No Heap Array support?

[Swoboderz]

trafficstars

you have text1b(), which would be the closest thing to char array support, though generic heap arrays arent supported at all, not even an option like

s.ext(ObjectPtr, PointerOwner{PointerType::NotNull}, ObjectCount);

[Swoboderz]

Do correct me if im wrong, but it seems like something that should really be supported.

[fraillt]

You're right. Currently, pointers as arrays are not supported :) I'm planning to implement them in future, but cannot tell exact date yet, but pull requests are welcome :)

[VelocityRa]

Any progress on this?

I have a void* of a specific size (known at runtime). It's already allocated before (de)serializing, I only need the data memcpy in/out of there. How can this be done?

I tried this as a quick and dirty way:

        if (_ptr) {
            for (auto i = 0 ; i < _size; ++i)
                s.value1b(*((U8*)_ptr + i));
        }

but my data is many GBs so this is (understandably) way too slow.

Edit: The cereal equivalent would be cereal::binary_data.

[fraillt]

Hi, You can do it two ways,

1. Write an extension.
2. Write wrapper type for ptr and size and implement ContainerTraits for it.

I would prefer to write an extension, it has more customization and less code to write. Here is an example:

namespace bitsery {

    namespace ext {

        template <typename TSize>
        class BufferData {
        public:

            explicit BufferData(TSize& size):_size{size} {}

            template<typename Ser, typename T, typename Fnc>
            void serialize(Ser &ser, const T &obj, Fnc &&fnc) const {
                auto& writer = ser.adapter();
                writer.template writeBytes<4>(static_cast<uint32_t>(_size));
                writer.template writeBuffer<1>(static_cast<const uint8_t*>(obj), _size);
            }

            template<typename Des, typename T, typename Fnc>
            void deserialize(Des &des, T &obj, Fnc &&fnc) const {
                auto& reader = des.adapter();
                uint32_t s=0u;
                reader.template readBytes<4>(s);
                _size = static_cast<TSize>(s);
                reader.template readBuffer<1>(static_cast<uint8_t*>(obj), _size);
            }

        private:
            TSize& _size;
        };
    }

    namespace traits {
        template<typename TSize, typename T>
        struct ExtensionTraits<ext::BufferData<TSize>, T> {
            using TValue = void;
            static constexpr bool SupportValueOverload = false;
            static constexpr bool SupportObjectOverload = true;
            static constexpr bool SupportLambdaOverload = false;
        };
    }

}

struct MyStruct {
    void* ptr;
    int ptr_size;
};

template <typename S>
void serialize(S& s, MyStruct& o) {
    s.ext(o.ptr, bitsery::ext::BufferData<int>(o.ptr_size));
}

[VelocityRa]

@fraillt That's very helpful, thanks!

[VelocityRa]

For our use case we don't even need to store the size. It doesn't change per-object. The code looks pretty simple to modify so I'll do that myself :)

Do you plan to merge this, btw?

[fraillt]

No, there are no plans to meet it, simply because it's easy to implement, but there might be too many edge cases that doesn't fit for everyone anyway. Like in your case, you don't need size;) but hope that this example will be useful for others as well:)

[VelocityRa]

Ok, though it's easy for you, you made the library :) For anyone else it'd take longer, for something that should be a basic feature (imo).

You can simply base it off of cereal::binary_data and it'll be fine. Also, I don't really see any other use cases besides size + no size. The "no size" alteration is optional and a lot of other people likely won't need it, so you can even omit it (or integrate it with a bool template parameter or something).

[fraillt]

I understand your view, and your points are valid. I try to cover most standard types, so at least std would work seamlessly, and for other things, I included what seamed necessary for me ;) and those that are not trivial to write (e.g. pointer support), or bitsery itself had to be modified to support them (growable). Basically idea is to provide necessary tools, so you could extend further. There are multiple things that might be useful, binary_data is one of them, the other is versioning and probably many more, what I want to do instead is maybe provide these as an example. I started some work here maybe I should write similar for binary_data as well, but for the moment, I have no plans to include them in the library.

[VelocityRa]

Ok :) Thanks for your work.

[Getshi]

Hi, I have a similar issue. I would like to serialize Eigen matrices. My use case will include std::vectors of hundreds of matrices with thousands of rows.

Given the examples above I only managed to get something working like

template <typename Ser, typename T, typename Fnc>
void serialize(Ser &ser, const T &matrix, Fnc &&fnc) const
{
  uint32_t rows = matrix.rows();
  uint32_t cols = matrix.cols();
  uint32_t elems = rows * cols;

  auto &writer = ser.adapter();
  writer.template writeBytes<4>(static_cast<uint32_t>(rows));
  writer.template writeBytes<4>(static_cast<uint32_t>(cols));
  for (uint32_t i = 0; i < elems; ++i)
    ser.value4b(matrix.data()[i]);
}

I had to resort to the (according to comments above) slow version of writing separate values, because writeBuffer requires integer input. I tried throwing away platform-independence by using a reinterpret_cast but that didn't work (I assume because it might require a different provided buffer size). Anyway, do you have suggestions for how to solve this nicely? FYI, the matrix.data() method exposes an underlying contiguous float*array. Maybe there is something different that I could do with container?

[Getshi]

Actually I think I got it to work with writeBuffer:

template <typename Ser, 
typename Scalar, int _Rows, int _Cols, int _Options, int _MaxRows, int _MaxCols, 
typename Fnc>
void serialize(Ser &ser, const ::Eigen::Matrix<Scalar, _Rows, _Cols, _Options, _MaxRows, _MaxCols> &matrix, Fnc &&fnc) const
{
  uint32_t rows = matrix.rows();
  uint32_t cols = matrix.cols();
  uint32_t elems = rows * cols;

  auto &writer = ser.adapter();
  writer.template writeBytes<4>(static_cast<uint32_t>(rows));
  writer.template writeBytes<4>(static_cast<uint32_t>(cols));

  static_assert(details::IsFundamentalType<Scalar>::value, "Value must be integral, float or enum type.");
  using TValue = typename details::IntegralFromFundamental<Scalar>::TValue;
  writer.template writeBuffer<sizeof(TValue)>(reinterpret_cast<const TValue *>(matrix.data()), elems);
}

This might be a stupid question and the wrong place to ask, but I see that writeBuffer then basically does a for loop to write each value - is this efficient / is there a way to code it such that it attempts to copy the entire buffer at once?

[fraillt]

Hello, I'm glad you make it work, I assume you don't need to deserialize this. Regarding writeBuffer, it will only iterate though every value if your platform's endianess doesn't match with bitsery config (default is little endian). Otherwise it will call std::copy_n, which in turns calls std::copy, which in most know compilers calls memmove. So unless you are using exotic hardware or compiler, it should be very efficient.

[Getshi]

Ok thanks for the clarification! I will be deserializing it too, I just didn't post the code since it's trivially the same, basically just replacing writeBytes and writeBuffer for readBytes and readBuffer. Out of curiosity, is the version I posted above platform independent? I just tried replicating what I saw in the bitsery code.

[fraillt]

If matrix element type is float, double or one of new fixed type int_32t, int_64t etc, then it's ok. BTW, preferred way to do serialization and deserialization would be to use extensions. This would allow to define one serialize function that would be less error prone solution.

[Getshi]

Thanks, that's what I did. I'm posting the full code here in case it might help someone.

Click to expand for full code (serializing dense Eigen matrices)

#ifndef BITSERY_EXT_EIGEN_H
#define BITSERY_EXT_EIGEN_H

#include "../traits/core/traits.h"
#include "../details/adapter_common.h"
#include "../details/serialization_common.h"
#include <Eigen/Dense>
#include <Eigen/Core>

namespace bitsery
{
  namespace ext
  {
    namespace Eigen
    {
      class Matrix
      {
      public:
        template <typename Ser,
                  typename Scalar, int _Rows, int _Cols, int _Options, int _MaxRows, int _MaxCols,
                  typename Fnc>
        void serialize(Ser &ser, const ::Eigen::Matrix<Scalar, _Rows, _Cols, _Options, _MaxRows, _MaxCols> &matrix, Fnc &&fnc) const
        {
          uint32_t rows = matrix.rows();
          uint32_t cols = matrix.cols();
          uint32_t elems = rows * cols;

          auto &writer = ser.adapter();
          writer.template writeBytes<4>(static_cast<uint32_t>(rows));
          writer.template writeBytes<4>(static_cast<uint32_t>(cols));

          static_assert(details::IsFundamentalType<Scalar>::value, "Value must be integral, float or enum type.");
          using TValue = typename details::IntegralFromFundamental<Scalar>::TValue;
          writer.template writeBuffer<sizeof(TValue)>(reinterpret_cast<const TValue *>(matrix.data()), elems);
        }

        template <typename Des,
                  typename Scalar, int _Rows, int _Cols, int _Options, int _MaxRows, int _MaxCols,
                  typename Fnc>
        void deserialize(Des &des, ::Eigen::Matrix<Scalar, _Rows, _Cols, _Options, _MaxRows, _MaxCols> &matrix, Fnc &&fnc) const
        {
          auto &reader = des.adapter();
          uint32_t rows = 0u, cols = 0u;
          reader.template readBytes<4>(rows);
          reader.template readBytes<4>(cols);
          uint32_t elems = rows * cols;

          matrix.resize(rows, cols);
          static_assert(details::IsFundamentalType<Scalar>::value, "Value must be integral, float or enum type.");
          using TValue = typename details::IntegralFromFundamental<Scalar>::TValue;
          reader.template readBuffer<sizeof(TValue)>(reinterpret_cast<TValue *>(matrix.data()), elems);
        }

      private:
      };

    } // namespace Eigen
  }   // namespace ext

  namespace traits
  {
    template <typename T>
    struct ExtensionTraits<ext::Eigen::Matrix, T>
    {
      using TValue = void;
      static constexpr bool SupportValueOverload = false;
      static constexpr bool SupportObjectOverload = true;
      static constexpr bool SupportLambdaOverload = false;
    };
  } // namespace traits

} // namespace bitsery

#endif //BITSERY_EXT_EIGEN_H

Usage: e.g. ser.ext(matrix, ext::Eigen::Matrix{});

[fraillt]

Thanks for sharing!