[UPDATED] with alignment. I wrote an extension you can now deserialize buffer without any copying.

[Diarica]

trafficstars

Hopefully useful for you! Originally, if you're using std::vector as a serialize field, you must get memory copying, it's annoying. And for some cases (game engine assets etc.) you definitely don't want any copying occured.

and if you would like to free the copied vector memory, it's typically hard in C++, because standard not say anything about how compliers should free the memory. in my cases, a 70mb asset file, deserialize it directly eat my over 120mb memory, it's unacceptable, it can direcly use original file buffer.

So I dive into BufferAdapter and Extension, I found it actually very easy to eliminate it copying behavior.

The only things you need, is a custom adapter and an extension.

with caution : There is only be single alignment value available for all the noncopyable buffers(otherwise alignment value is multiple of others buffer like 32, 16, 8, 4), this might need some safety improvements(currently it's allow each buffer to say a wanted alignment) The deserialization buffer(your passed into deserialize), MUST BE ALIGNED AS SAME AS noncopyable buffer, otherwise nothing will actually aligned.

Usage :

template <typename S>
   void serialize(S& s)
   {

       
       s.ext(ProcessedAssetData, bitsery::ext::BufferDataNoCopying<uint32_t>(ProcessedAssetDataSize,32));
       
       s.ext(SerializationData, bitsery::ext::BufferDataNoCopying<uint32_t>(SerializationDataSize,32));
       
       s.text1b(FactoryName,40);
       s.text1b(AssetName,1023);
   
   }

namespace bitsery {
namespace ext {

template <typename TSize>
class BufferDataNoCopying
{
public:

  explicit BufferDataNoCopying(TSize& size, uint16_t alignment):_size{size},_alignment(alignment) {}

  template<typename Ser, typename T, typename Fnc>
  void serialize(Ser &ser, const T &obj, Fnc &&fnc) const {
    auto& writer = ser.adapter();
    writer.template writeBytes<4>(static_cast<uint32_t>(_size));

    
    auto dummyNeed = writer.currentWritePos() % _alignment;
    
    if (dummyNeed != 0)
    {
      dummyNeed = _alignment - dummyNeed;
      // Because we need to copy these meaningless data, so just point to an random address and copy these data.
      // Avoid any extra allocation.
      writer.template writeBuffer<1>(((const uint8_t*)&writer), dummyNeed);
    }
    
    writer.template writeBuffer<1>(static_cast<const uint8_t*>(obj), _size);
  }

  template<typename Des, typename T, typename Fnc>
  void deserialize(Des &des, T &obj, Fnc &&fnc) const {
    auto& reader = des.adapter();
    
    reader.template readBytes<4>(_size);

    
    auto dummyNeed = reader.currentReadPos() % _alignment;
    
    if (dummyNeed != 0)
    {
      dummyNeed = _alignment - dummyNeed;
      //Skip the meaningless.
      reader.currentReadPos(reader.currentReadPos() + dummyNeed);

    }
    reader.readBufferNoCopying((obj), _size);

    
  }

private:
  TSize& _size;
  uint16_t _alignment;
};
}


namespace traits {
template<typename TSize, typename T>
struct ExtensionTraits<ext::BufferDataNoCopying<TSize>, T>
{
  using TValue = void;
  static constexpr bool SupportValueOverload = false;
  static constexpr bool SupportObjectOverload = true;
  static constexpr bool SupportLambdaOverload = false;
};
}

}

[fraillt]

I'm glad it worked for you and thanks for sharing! BTW, did you looked at @victorstewart #120, maybe it could make things for you even simpler?

[Diarica]

I'm glad it worked for you and thanks for sharing! BTW, did you looked at @victorstewart #120, maybe it could make things for you even simpler?

Okay, It's do make things simpler, but it's do modified the source code. I think copying behavior should only be decided by user. Because some data is no need to avoid copying, and user may free the buffer just after deserialization(causing the field being invalid). In my project, game engine asset pipeline, some binary blob data(textures, audio streams etc.) use original buffer, but for serializable fields they still copying.

My suggestion is, maybe you can provide a no copying read function in default input adapter, and simply make an built-in extension and let user decide.