Typed networking library

[vurvdev]

Considering the amount of people using net.writeTable in the discord I think there would be value in creating a library to type a predefined struct of data in order to read/write from a net message to save bandwidth but have a more convenient alternative to manually writing tables.

local Student = net.Struct [[
    name: cstr, -- null terminated string
    gpa: f32
]]

local Classroom = net.Struct ([[
    n_students: u32,
    students: [Student; $n_students] -- explicit vector, probably also want an implicit version (provide number type for length of vector)
]], { Student = Student })

-- server
local bytes = Classroom:encode {
    n_students = 2,
    students = {
        { name = "Bob", gpa = 2.3 },
        { name = "Joe", gpa = 3.4 }
    }
}

net.start("lame")
    net.writeUInt(#bytes, 32)
    net.writeData(bytes, #bytes)
    -- or
    net.writeStruct(Classroom, { ... }) -- like this the most
    -- or
    Classroom:writeNet({ ... })
net.send()

-- client
net.receive("lame", function(ply, len)
    local classroom = Classroom:decode(net.readData(net.readUInt(32)))
    -- or
    local classroom = net.readStruct(Classroom)
    -- or
    local classroom = Classroom:readNet()
end)

My datastream library already does this, except for having recursive structs / custom types.

I don't think it'd be too much work implementing this considering stringstream already exists. This is just implementing a basic lua pattern parser and code generator. Maybe this should be a part of stringstream rather than net, though

Why builtin

There really isn't much reason for this being builtin besides having wider outreach / ease of access, which helps when the target audience are those who want the convenience of net.Read/WriteTable but without the heavy net usage.

[thegrb93]

This could get very complicated for something that's easy to do with just lua.

[vurvdev]

Open to whatever syntax will make it as simple as possible, if that's what you're getting at

Right now it's based off Rust since there's less ambiguity for parsing. Maybe this alternate syntax:

net.Struct([[
    int32 foo,
    vec8[i32] bar, -- vector w/ u8 length
    vec[i32] baz, -- maybe a default length vector (defaults to u16?)
    special custom,
]], { special = ... })

[vurvdev]

This could get very complicated for something that's easy to do with just lua.

Easy to do for very small structs maybe but when you get to networking lists of items it gets annoying

[thegrb93]

I prefer a simple paradigm like this, which also lets you add conditionals or functionality to your write/read functions (for further size savings if needed).

local Student = class("Student")
function Student:initialize(name, gpa)
	self.name = name
	self.gpa = gpa
end
function Student:writeData(ss)
	ss:writeString(self.name)
	ss:writeFloat(self.gpa)
end
function Student:readData(ss)
	self.name = ss:readString()
	self.gpa = ss:readFloat()
end


local Classroom = class("Classroom")
function Classroom:initialize(students)
	self.students = students
end
function Classroom:writeData(ss)
	ss:writeArray(self.students, function(s) s:writeData(ss) end)
	return ss
end
function Classroom:readData(ss)
	self.students = ss:readArray(function() local s = Student:new() s:readData(ss) return s end)
end


local classroom = Classroom:new({
	Student:new("Bob", 2.3),
	Student:new("Joe", 3.4)
})

net.start("lame")
	local data = classroom:writeData(bit.stringstream()):getString()
	net.writeUInt(#data, 32)
	net.writeData(data, #data)
net.send()

-- client
net.receive("lame", function(ply, len)
	local classroom = Classroom:new()
	classroom:readData(bit.stringstream(net.readData(net.readUInt(32)))
end)

[vurvdev]

Indeed but that is still a lot more effort, when this is targeting the group that wants to avoid that and just uses net.write/readTable. Also needs you to use middleclass/OOP

[thegrb93]

Same argument goes for this, which requires learning new syntax and setting up the struct dependencies. Are newbies really going to prefer doing that?

Also, you don't have to use middleclass/oop, it's just cleaner looking with it. I wouldn't consider that a downside.

[vurvdev]

Same argument goes for this, which requires learning new syntax and setting up the struct dependencies.

As long as the syntax is simple it shouldn't have to be "learned", just <type> <name> every line (or <name>: <type>?), int<bits>/uint<bits>, vec<bits>[<item>] as the most complex type.

Are newbies really going to prefer doing that?

Newbies will never prefer anything over writeTable since it's so easy. What's nice about this is it's practically plug and play, just create a small struct definition at the top and replace write/readTable with write/readStruct.

So we could refer users to this if they use writeTable as a way to improve it without having to largely rewrite their code

[thegrb93]

Maybe you could ask in the discord to gauge interest? I guess it can be builtin so long as it's not much longer than the doc parser code.

[thegrb93]

I saw Name's idea about making the struct out of lua tables instead of the new syntax, which sounds a lot more feasible. I wouldn't be against adding that.

[vurvdev]

I did raise my concerns about it. it would either require weird namespacing or having a global for every type:

local Tuple, Vec, i32, String = net.Struct.Tuple, net.Struct.Vec, net.Struct.i32, net.Struct.String

local MyThing = Tuple(
  Vec(i32),
  String
)

net.start("foo")

net.writeStruct(MyThing, {
  { 1, 2, 3 },
  "test"
})

net.send()

net.receive("foo", function()
  local thing = net.readStruct(MyThing)
end)

Feel like it's more of a burden, but I suppose could be done

[thegrb93]

Or

local st = net.Struct

local MyThing = st.Tuple(
  st.Vec(st.i32),
  st.String
)

[vurvdev]

---@enum Variant
local Variant = {
	UInt = 1,
	Int = 2,
	Bool = 3,
	Float = 4,
	Double = 5,
	CString = 6,

	List = 7,
	Struct = 8,
	Tuple = 9,

	Entity = 10,
	Player = 11,
	Angle = 12,
	Vector = 13
}

---@class NetObj
---@field variant Variant
---@field data any
local NetObj = {}
NetObj.__index = NetObj

function NetObj.UInt(bits --[[@param bits integer]])
	return setmetatable({ variant = Variant.UInt, data = bits }, NetObj)
end

function NetObj.Int(bits --[[@param bits integer]])
	return setmetatable({ variant = Variant.Int, data = bits }, NetObj)
end

NetObj.Int8 = NetObj.Int(8)
NetObj.Int16 = NetObj.Int(16)
NetObj.Int32 = NetObj.Int(32)

NetObj.UInt8 = NetObj.UInt(8)
NetObj.UInt16 = NetObj.UInt(16)
NetObj.UInt32 = NetObj.UInt(32)

NetObj.Float = setmetatable({ variant = Variant.Float }, NetObj)
NetObj.Double = setmetatable({ variant = Variant.Double }, NetObj)

NetObj.Str = setmetatable({ variant = Variant.CString }, NetObj)

function NetObj.Tuple(... --[[@vararg NetObj]])
	return setmetatable({ variant = Variant.Tuple, data = { ... } }, NetObj)
end

function NetObj.List(ty --[[@param ty NetObj]], bits --[[@param bits integer?]])
	return setmetatable({ variant = Variant.List, data = { ty, bits or 16 } }, NetObj)
end

function NetObj.Struct(struct --[[@param struct table<string, NetObj>]])
	return setmetatable({ variant = Variant.Struct, data = struct }, NetObj)
end

NetObj.Entity = setmetatable({ variant = Variant.Entity }, NetObj)
NetObj.Player = setmetatable({ variant = Variant.Player }, NetObj)

function NetObj:write(value  --[[@param value any]])
	if self.variant == Variant.Tuple then ---@cast value integer[]
		for i, obj in ipairs(self.data) do
			obj:write( value[i] )
		end
	elseif self.variant == Variant.UInt then
		net.WriteUInt(value, self.data)
	elseif self.variant == Variant.Int then
		net.WriteInt(value, self.data)
	elseif self.variant == Variant.Bool then
		net.WriteBool(value)
	elseif self.variant == Variant.Float then
		net.WriteFloat(value)
	elseif self.variant == Variant.Double then
		net.WriteDouble(value)
	elseif self.variant == Variant.CString then
		net.WriteString(value)
	elseif self.variant == Variant.List then
		local len, obj = #value, self.data[1]

		net.WriteUInt(len, self.data[2])
		for i = 1, len do
			obj:write( value[i] )
		end
	elseif self.variant == Variant.Struct then
		for key, obj in SortedPairs(self.data) do
			obj:write( value[key] )
		end
	elseif self.variant == Variant.Entity then
		net.WriteEntity(value)
	elseif self.variant == Variant.Player then
		net.WritePlayer(value)
	elseif self.variant == Variant.Angle then
		net.WriteAngle(value)
	elseif self.variant == Variant.Vector then
		net.WriteVector(value)
	end
end

function NetObj:read()
	if self.variant == Variant.Tuple then
		local items, out = self.data, {}
		for i, item in ipairs(items) do
			out[i] = item:read()
		end
		return out
	elseif self.variant == Variant.UInt then
		return net.ReadUInt(self.data)
	elseif self.variant == Variant.Int then
		return net.ReadInt(self.data)
	elseif self.variant == Variant.Bool then
		return net.ReadBool()
	elseif self.variant == Variant.Float then
		return net.ReadFloat()
	elseif self.variant == Variant.Double then
		return net.ReadDouble()
	elseif self.variant == Variant.CString then
		return net.ReadString()
	elseif self.variant == Variant.List then
		local out, obj = {}, self.data[1]
		for i = 1, net.ReadUInt(self.data[2]) do
			out[i] = obj:read()
		end
		return out
	elseif self.variant == Variant.Struct then
		local out = {}
		for key, obj in SortedPairs(self.data) do
			out[key] = obj:read()
		end
		return out
	elseif self.variant == Variant.Entity then
		return net.ReadEntity()
	elseif self.variant == Variant.Player then
		return net.ReadPlayer()
	elseif self.variant == Variant.Angle then
		return net.ReadAngle()
	elseif self.variant == Variant.Vector then
		return net.ReadVector()
	end
end

-- example

local t = NetObj.Tuple(
	NetObj.List(NetObj.UInt8, 8),
	NetObj.Str,
	NetObj.Struct {
		foo = NetObj.Double,
		bar = NetObj.Str
	}
)

net.Start("net_thing")
	t:write {
            { 1, 2, 7, 39 },
            "foo bar",
            {
                foo = 239.1249,
                bar = "what"
            }
        }
net.Broadcast()