Buffer length checking when writing to a raw buffer

[stephenberry]

Originally requested in: #1281

It would be helpful to have another option to check the length when writing to a raw buffer and return an error if the length of the buffer were to be exceeded.

[notable-equivalent]

So just to clarify, is it currently not possible to safely write JSON to a buffer without either a) using heap memory (so that we could use something resizable) or b) having exceptions enabled (so that we could use something like std::inplace_vector whose resize method throws when the capacity is exceeded)?

This may mean that the library is unfortunately not suitable for use on embedded devices yet, because those are pretty hard requirements for a lot of projects.

[stephenberry]

@notable-equivalent, I was recently working on this. You're correct that resize can only fail with exceptions. I use exceptions for out of memory issues on embedded hardware, but I realize this isn't suitable for many others. If you use the standard template library then exceptions turn into aborts with fnoexcept.

You can currently use a stack allocated std::basic_string or your own stack allocated (or fixed heap) wrapper that matches the resizable buffer concept, but your error handling would need to abort or kill the thread if not using exceptions.

This is a serious limitation, and one that I've been thinking about for a while, but haven't had anyone request it.

This particular issue was considering just a raw/fixed buffer approach, but I'm thinking a failable buffer approach would be better and more flexible, while still allowing a raw/fixed buffer to be used under a wrapper. The failable resize could just require the resize function to return a boolean for success/failure.

What kind of container types do you use for buffers in your code?

[notable-equivalent]

It's mainly raw buffers, usually just a std::array. I think accepting a raw buffer and a size (or equivalently, a std::span or similar) would cover most use cases.

But perhaps it makes sense internally to use a more general interface, maybe a try_resizable concept for types with a bool try_resize(std::size_t) method, since it's easier to extend in the future. And then spans can just be wrapped internally in something like:

class span_buffer_wrapper : public std::span<char>
{
public:
    std::size_t size() const { return this->my_span.size(); }
    bool try_resize(std::size_t n) { return n <= this->size(); }
    char& operator[](std::size_t n) { return this->my_span[n]; }
private:
     std::span<char> my_span;
};

Although this interface seems like something that people would often want to implement on external library types, since nothing really has a try_resize method, so maybe it makes more sense to do a traits-like implementation:

template<>
struct glz::buffer_traits<std::span<char>>
{
    std::size_t size(std::span<char> const& self) { return self.size(); }
    bool try_resize(std::span<char>& self, std::size_t n) { return n <= self.size();  }
};

---

As an aside, I listened to Khalil Estell's excellent talk, and I definitely am looking into using exceptions for embedded projects. But the ecosystem doesn't quite seem production-ready yet, especially for RISC-V

[stephenberry]

Thanks for your feedback, I really like this buffer_traits idea. I'm not sure how long it will be to get this implemented, but it will be a higher priority.