red5-server icon indicating copy to clipboard operation
red5-server copied to clipboard

DataTypes isComplexType fix

Open kdkd opened this issue 1 month ago • 2 comments

Commit from kdkd fork e514d5f

io/object/DataTypes.java:

isComplexType uses type >= CORE_ARRAY || type <= CORE_OBJECT, which incorrectly returns true for nearly all types; basic types are misclassified as complex, leading to incorrect serialization/deserialization flow for most values.

Change || to &&


kdkd avatar Nov 26 '25 19:11 kdkd

What is the correct behavior here? I'm a little blurry on what it's trying to accomplish. With it changed to && this is the truth table for this function:

Type name Hex Dec isComplexType Reason
CORE_SKIP 0x00 0 false Below CORE_ARRAY lower bound
CORE_NULL 0x01 1 false Below CORE_ARRAY lower bound
CORE_BOOLEAN 0x02 2 false Below CORE_ARRAY lower bound
CORE_NUMBER 0x03 3 false Below CORE_ARRAY lower bound
CORE_STRING 0x04 4 false Below CORE_ARRAY lower bound
CORE_DATE 0x05 5 false Below CORE_ARRAY lower bound
CORE_ARRAY 0x06 6 true At CORE_ARRAY lower bound
CORE_MAP 0x07 7 true Within 0x06–0x09 range
CORE_XML 0x08 8 true Within 0x06–0x09 range
CORE_OBJECT 0x09 9 true At CORE_OBJECT upper bound
CORE_BYTEARRAY 0x10 16 false Above CORE_OBJECT upper bound
OPT_REFERENCE 0x11 17 false Above CORE_OBJECT upper bound
CUSTOM_MOCK_MASK 0x20 32 false Above CORE_OBJECT upper bound
CUSTOM_AMF_MASK 0x30 48 false Above CORE_OBJECT upper bound
CUSTOM_RTMP_MASK 0x40 64 false Above CORE_OBJECT upper bound
CUSTOM_JSON_MASK 0x50 80 false Above CORE_OBJECT upper bound
CUSTOM_XML_MASK 0x60 96 false Above CORE_OBJECT upper bound
CORE_END_OBJECT 0xFF 255 false Above CORE_OBJECT upper bound

How is this supposed to work?

kdkd avatar Nov 28 '25 23:11 kdkd

CORE_ARRAY 0x06 6 true At CORE_ARRAY lower bound CORE_MAP 0x07 7 true Within 0x06–0x09 range CORE_XML 0x08 8 true Within 0x06–0x09 range CORE_OBJECT 0x09 9 true At CORE_OBJECT upper bound

This is where we'll run into dragons, with this being legacy AMF, I can only look a the statement to discern what was implied. The way I read the original it looks like only these 4 types would pass, but the way its written is confusing and sub-optimal IMHO; if I were to do this today, I'd probably use a static array and check for membership or being lazy use a few "||" in a row.

mondain avatar Nov 29 '25 16:11 mondain