Cannot deserialize..

[Eldevia]

Hi, i am getting this error, i tried using different .sav files but it's the same, though offsets do change from each file, the offset numbers are still strange and i have no clue why they are. If you could help I would be the happiest!

throw new ERR_OUT_OF_RANGE(type || 'offset', RangeError [ERR_OUT_OF_RANGE]: The value of "offset" is out of range. It must be >= 0 and <= 2463. Received -947025976

[Eldevia]

If you help us resolve this problem asap, we can compensate your time helping our team <3

[ch1pset]

Would you be able to provide me the specific .sav you are trying to deserialize?

Also, when deserializing, the Serializer created a buffer that was sized 2463 bytes. The offset being a weird value might just mean that the code is expecting a certain structure, or variation of a specific structure, but is actually reading something that I never saw in BPM game saves. There could be some padding that isn't being accounted for.

[Eldevia]

Thank's for the reply <3 Here's the .sav files I am using from the game. savfiles.zip

[ch1pset]

So I know, at least the first problem. The property LastPlayerLocation is a StructProperty that contains a single Vector, which is not implemented because I did not encounter Vector with BPM. Looking at the binary data in HxD, it looks like Vector is a data structure that contains 3 values that are 4 bytes each(I'm assuming these are floats).

Also, this StructProperty and the next one(LastPlayerRotation) don't seem to conform to the StructProperty structure I've implemented. I THINK it's because they only contain a single property, there isn't a list that terminates with None in these 2 structs in the save file.

This should be a fairly easy fix, just gotta implement Vector and Rotator, and find a way to not have the Tuple in the StructProperty if it only contains 1 property.

EDIT This is the Rotator I was talking about:

Also, I will need to look at these structs a bit more closely, at a glance, I'm not exactly sure how to conditionally make these work with the existing implementation.

EDIT 2 Created branch issue-7-fix for this. I should have time to work on it this week. Although, feel free to work on it yourself in a fork and just PR any working changes(that don't break anything of course).

[Eldevia]

Thank you. We have lots of experience on Javascript and Node, but we are very fresh on binary stuff, could you maybe say what we have to do to fix? We then would try to follow your instructions, if we succeed fixing this we will of course give the source code. Also not sure whats vector and rotator in binary world.

[ch1pset]

So the structure of the LastPlayerRotation is as follows(NOTE: LastPlayerLocation structure is identical):

Bytes	Description	Value
13 00 00 00	Size of Name string	19 bytes
4C 61 73 74 50 6C 61 79 65 72 52 6F 74 61 74 69 6F 6E 00	Name string	"LastPlayerRotation\0"
0F 00 00 00	Size of Type string	15 bytes
53 74 72 75 63 74 50 72 6F 70 65 72 74 79 00	Type string	"StructProperty\0"
0C 00 00 00	Size of Stored data	12 bytes
00 00 00 00	Padding	4 null characters
08 00 00 00	Size of StoredType string	8 bytes
52 6F 74 61 74 6F 72 00	StoredType string	"Rotator\0"
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00	Padding	17 null characters
00 00 00 00 B0 5D 32 C3 00 00 00 00	Stored Data	[ 0, -178.36..., 0 ]

You can see, the data highlighted in this image:

The 17 bytes of padding is standard in all StructProperty structures, and the stored data always begins after this padding. So you can just count the bytes read from that point until the end of the 12 bytes. The problem is, the way I've implemented StructProperty involves a class called Tuple that I created to handle lists(which end with None in the save file). The reason I did this, was that BPM only had StructProperty that were structured that way. So a solution to that might involve rewriting either the StructProperty class or the Tuple class, but preferably, just adding some conditional statement somewhere to handle this particular case.

Aside from that, Vector and Rotator need to be added as classes that extend from Property, and implement the inherited functions. I'm not entirely familiar with Vector or Rotator but I'm making the assumption that they are just arrays with 3 floats, since the numbers look reasonable in HxD editor as floats. Also, we can see that the size of the stored data is 12 bytes, and it's reasonable to assume it's divided into 4 bytes either as int32 or float. For Rotator, it might also be the case that 4 bytes out of those 12 are actually just padding, and there are only 2 4byte values, or 8 bytes of padding and a single 4 byte value. I'm not entirely sure, but it wouldn't hurt to interpret that as 3 values in an array.

Here, you can analyze a BPM save using the tool if it would help you understand how the binary data is formatted ContinueStateV2.zip

Also, every new Property must implement serialize() and deserialize(), you can use a Buffer for serialize() but deserialize() receives a Serializer object. I created the Serializer class to handle keeping track of the current offset automatically, and it just makes reading/writing values easier. I kind of wish I exclusively used Buffer, but it would have resulted in a lot of additional boilerplate.

EDIT There is a little bit more information regarding specific implementation of Property in the README for this repo. I tried to include as much documentation as possible, but I'm only 1 person so a significant amount of information is missing.

[Eldevia]

It is a lot of information and new terms to process, I will learn some stuff. If i am lost i will get back to you! Thank you very much again

[ch1pset]

So I'm still looking into a good solution for deserializing your .sav files. I was able to produce an output by pretty much ignoring the StructProperty data. I think I might have some ideas on how to make this work:

1. The Tuple class should be modified so that if only 1 property is added, the "None" string is not expected in the save file
2. The Vector and Rotator classes should have a normal Property structure, but in these particular cases be compressed to just the 12 bytes of data within the Tuple of the StructProperty called LastPlayerLocation and LastPlayerRotation

Here is the output file I was able to produce: output1.zip

And the changes to StructProperty.deserialize() I made to produce it:

    deserialize(serial, size) {
        console.log(`Deserializing ${this.Name} Size: ${size}`)
        serial.seek(4);
        this.StoredPropertyType = serial.readString();
        serial.seek(17);
        // let end = serial.tell + size;
        // let i = 0;
        let data_buf = serial.read(size);
        console.log(data_buf);
        // while (serial.tell < end) {
        //     let Name = this.StoredPropertyType;
        //     let Type = 'Tuple';
        //     let prop = PropertyFactory.create({ Name, Type })
        //     prop.deserialize(serial)
        //     this.Properties.push(prop);
        //     i++;
        // }
        console.log(`Done Deserializing ${this.Name} Offset: ${serial.tell}`)
        return this;
    }

EDIT Hmm, looking at Tuple, I think that class is just fundamentally incompatible with this case. I don't like it, but the only reasonable solution I can think of is to check for Vector and Rotator within the StructProperty.deserialize() function. A more robust solution would involve refactoring a significant amount of the codebase to remove the need for Tuple... which technically would be the better thing to do since Tuple is not exactly necessary. I made it to encapsulate lists of properties that were delimited with None in the save files. It serves no functional purpose really, other than to work nicely with the object oriented implementation.

[Eldevia]

Hi it seems like im getting some progress. But not sure how to use seek(), and how to get the Stored Data from the serial

[ch1pset]

Serializer.seek(bytes) adds the given number of bytes to the current offset in the data buffer and sets the offset to the resulting value:

seek(bytes) {
    this._offset += bytes // bytes is a Number
}

To return the data buffer itself from the Serializer object, you can use the Data attribute:

let data_buf = serial.Data // returns a nodejs Buffer

OR, if the data is a sub-part of the data buffer being deserialized, you should use Serializer.read(bytes):

let data_buf = serial.read(bytes_to_read) // returns a Buffer

[Eldevia]

Also why AnimalSkin doesnt have any properties?

[ch1pset]

This section of the code I commented out in StructProperty.deserialize() would handle that for AnimalSkin, but it doesn't work on LastPlayerLocation and LastPlayerRotation:

        // let end = serial.tell + size;
        // while (serial.tell < end) {
        //     let Name = this.StoredPropertyType;
        //     let Type = 'Tuple';
        //     let prop = PropertyFactory.create({ Name, Type })
        //     prop.deserialize(serial)
        //     this.Properties.push(prop);
        // }

Once we can store Vector and Rotator data into an object, we can think about bringing back this functionality, but in a way that doesn't break anything. I'm just trying to focus on one thing at a time.

[Eldevia]

I've made that Vector and Rotator would work in StructProperty. Now i need to find a way to get the commented code back

[Eldevia]

i have uncommented code back. It works good, but doesnt store AnimalSkin properties

[ch1pset]

I see, I wrote a semi-generic VectorProperty class for both Vector and Rotator and can see SkinName is a NameProperty, which might be similar to StrProperty. Let me look a bit closer at the binary.

[ch1pset]

Okay, so here is the structure of SkinName:

Bytes	Description	Value
09 00 00 00	Property Name Size	9 bytes
53 6B 69 6E 4E 61 6D 65 00	Property Name	"SkinName\0"
0D 00 00 00	Type String Size	13 bytes
4E 61 6D 65 50 72 6F 70 65 72 74 79 00	Type String	"NameProperty\0"
0D 00 00 00	Property Data Size	13 bytes
00 00 00 00 00	Padding	5 null characters
09 00 00 00	Data string size	9 bytes
45 6C 65 70 68 61 6E 74 00	Data string	"Elephant\0"

The Property Data Size is the total size of the Data string + Data string size in bytes, so 4 + 9 = 13 bytes;

[ch1pset]

Yeah, so this is the same structure as StrProperty, you can just add NameProperty as an alias to PropertyFactory, let me see if I can get that working.

[ch1pset]

I was able to get it working with 1 new line in the index.js that contains the aliases for properites for PropertyFactory output1.zip

[ch1pset]

The 1 line I added was:

PropertyFactory.Properties['NameProperty'] = StrProperty;

EDIT NOTE: this doesn't work with Vector or Rotator in our case, but I would like a generic implementation that does so PropertyFactory can handle instantiation of Property type objects.

[ch1pset]

I've pushed a commit to the issue-7-fix branch. You can test it out.

[Eldevia]

How about serializing all this? In propertyfactory.js i added console.log(obj) before everything. And here it shows that it fails

[Eldevia]

for some reason it wants to grab the first property

[Eldevia]

heres the whole log log.txt

[ch1pset]

Ah, I see, I overlooked the StructProperty.from() static function. What's happening is, the script is trying to parse the json, and the from() function tries to create a property but the way I implemented, it needs a name and needs to be added as an alias to PropertyFactory... uh, should be simple to fix that.

[ch1pset]

I pushed a commit that fixes the serialization of VectorProperty which was the main issue there. I was able to produce an identical copy of the first .sav you provided with it: output1.zip