zenkit::VfsNode or zenkit::Read buffer access

[Katharsas]

I first need to explain a bit how file handling works in ZenRen (the code is not that important)

---

FileHandle is basically a variant of filepath or VFS node which i use to pass around file references:

// abstracts over real vs VFS files
struct FileHandle {
	const std::string name = "";
	const std::filesystem::path* path = nullptr;// -> if not null, this is a real file
	const zenkit::VfsNode* node = nullptr;// -> if not null, this is a ZenKit VFS entry
};

---

FileData is what i use to actually pass around actual data:

struct FileData {
public:
	// not copyable, must be moved
	FileData(const FileData&) = delete;
	FileData(FileData&&) noexcept;
	~FileData() noexcept;

	// backed by some static buffer
	FileData::FileData(const std::string& name, const std::byte * data, uint64_t size)
		: name(name), data(data), size(size) {}

	// backed by mmap which is kept open for this objects lifetime
	FileData::FileData(const std::string& name, const std::byte * data, uint64_t size, zenkit::Mmap&& mmap)
		: name(name), data(data), size(size), mmap(std::move(mmap)) {}

	// backed by heap buffer which is kept for this objects lifetime
	FileData::FileData(const std::string& name, const std::byte * data, uint64_t size, std::unique_ptr<std::vector<std::uint8_t>>&& buffer)
		: name(name), data(data), size(size), buffer(std::move(buffer)) {}

	// backed by heap buffer with unknown lifetime
	FileData::FileData(const std::string& name, const std::byte* data, uint64_t size, const std::shared_ptr<std::vector<std::uint8_t>>& buffer)
		: name(name), data(data), size(size), bufferShared(buffer) {}

	const std::string name;
	const std::byte * const data = nullptr;
	const uint64_t size = 0;
private:
	// std::unique_ptr (and also zenkit::Mmap) is not copyable, making this struct also not copyable
	// both are automatically deleted when this struct goes out of scope or is deleted

	// TODO consider using variant?
	// TODO wrap mmap in unique pointer?
	std::optional<zenkit::Mmap> mmap = std::nullopt;
	std::optional<std::unique_ptr<std::vector<std::uint8_t>>> buffer = std::nullopt;
	std::optional<std::shared_ptr<std::vector<std::uint8_t>>> bufferShared = std::nullopt;
};

As you can see, FileData is a mess and probably has lifetime problems that I don't care too much about, because when i create a FileData object I usually consume or parse it immediately, but it sure is not pretty.

---

Now I do like zenkit::Read, I think that std streams are horrible to work with directly and zenkit::Read really builds a nice abstraction over all kinds of streamable data.

However, the problem i have is that most of my code and libraries expect full buffer access when accessing resources, for example I am using DirectXTex to process my textures and of course

• it does not want zenkit::Read
• it wants std::byte * data, uint64_t size.

So I would really love to entirely replace the implementation of FileData by just wrapping a single zenkit::Read!

But it seems kind of awkward to get the size of the underlying data buffer out of zenkit::Read. I understand that not every implementation of zenkit::Read would even have a size, but in my case this is always true.

And using tell and then seek to seek the end of the buffer just so i can calculate the size seems very awkward.

So right now, i am constructing my FileData like this:

const FileData getData(const FileHandle handle)
{
	if (handle.path != nullptr) {
		auto mmap = zenkit::Mmap(*handle.path);
		return FileData(handle.name, mmap.data(), mmap.size(), std::move(mmap));
	}
	if (handle.node != nullptr) {
		phoenix::buffer buffer_view = handle.node->open();
		return FileData(handle.name, buffer_view.array(), buffer_view.limit());
	}
}

So i am relying on phoenix::buffer right now and cannot switch to a new version of ZenKit 😭

Now if this was Java and zenkit::Read was InputStream this would not be a problem because most libs support InputStream, but here every library just takes std::byte * data, uint64_t size, so i would like to ask if it would be possible to add a method to zenkit::Read that returns the underlying pointer+size when the implementation supports it.

Suggestion: std::span<std::byte> as_buffer() would make it clear its non-owning (but span is c++ 20).

Edit: Alternatively, a way to get a pointer + length form a zenkit VFS node would also work i guess. In fact it would probably be cleaner because not everybody needs a fully fledged interface for parsing stuff when they just want a data buffer from a VFS node.

[lmichaelis]

Hm, I can see why you'd need buffer access, though specifically with the new mmap-less I/O system that's not super easy to provide. I feel like the best solution is actually just a helper function which seeks to END, tells, then seeks to BEG again and reads the data into a vector<uint8_t> when required.

I would not be opposed to add a remaining or size function to Read.