lslib
lslib copied to clipboard
Norbyte
Reversing BG3 "NewAge"
Heya!
Since many hands make light work, I thought I'd share my first draft of reversing the NewAge LSMF format (Which I think stands for Larian Studios Metadata File).
It's a 010 Editor template, but it reads like C/C++ really.
struct Block0Header {
uint64 RelativeOffset;
uint64 BlockSize;
uint32 NameSize;
uint16 NumberOfNameIndexEntries;
byte Unknown1[10]<fgcolor=cRed>;
};
struct Block0Entry(uint64 namesPosition) {
uint64 Offset;
uint64 NameSize;
byte Unknown1[32]<fgcolor=cRed>;
local uint64 currentPosition = FTell();
FSeek(namesPosition + Offset);
char Name[NameSize];
FSeek(currentPosition);
};
struct Block0Body(Block0Header &header) {
local uint64 currentPosition = FTell();
local uint64 namesPosition = currentPosition + header.RelativeOffset;
FSeek(namesPosition + header.NameSize);
Block0Entry NameIndex(namesPosition)[block0Header.NumberOfNameIndexEntries]<optimize=false>;
FSeek(currentPosition);
};
struct Block1 {
local uint64 currentPosition = FTell();
byte Unknown1[8]<fgcolor=cRed>;
uint64 RelativeOffset;
byte Unknown2[8]<fgcolor=cRed>;
uint64 BlockSize;
byte Unknown3[8]<fgcolor=cRed>;
FSeek(currentPosition + RelativeOffset);
byte Unknown4[BlockSize-RelativeOffset];
};
struct LSMF {
char MagicHeader[4]<bgcolor=cLtGray>;
byte Unknown1[12]<fgcolor=cRed>;
Block0Header block0Header<bgcolor=cLtGray>;
Block0Body block0Body(block0Header)<bgcolor=cLtGray>;
Block1 block1<bgcolor=cLtGray>;
};
LSMF lsmf;
Obviously work in progress, but as this file gets shaped more and more, I reckon a lot of tech-savvy players can pitch in their thoughts on the unknown bits.
With this version, Block1 is still very WIP. Just by looking at the data, the current size seems alright. But I'm pretty sure it's not the full block size. As between the Block0 and Block 1, there is a chunk of space (about 1/4 of the NewAge data in size), that contains your character names (among other things). I'm pretty sure if we look for a similar string-lookup structure for Block0, that we will find pointers back to other parts in the file.
EDIT: By just looking at some of the data and recognizing certain structures from other games, I'm pretty sure there are is actually 3D model data in this file. Give-aways are usually the large repeated structures repeating the alphabet (IndexBuffer data). Like these:
I found a WebP in my NewAge, and it contains an image of Gale.
Not sure why, since Gale was not the active player at the time of saving. There are no other WebP files, the WebP is also not an animated version with more than 1 frame. I would expect to find the images of my other party members to, though.
EDIT: I was wrong, there is more than one RIFF header.
Ahh, I'm beginning to see some logic in this. So the list of block 0 entries appear to be references to components. The unknown bit of the entry references a type and an offset (not exactly sure what it's relative to). They are also sequential, so the next "component" starts where the last one ended.
For example if you take a look at the last few entries (enumeration values like EPriority, EState), there are flags describing 40, 32, 16 size, etc.
The components then store data about, for example, character creation.
EDIT:
I: Name entries
I: 0 = core.v0.Level (32)
I: 1 = core.v0.EntityId (275712)
I: 2 = game.action_resources.v1.Component (4128)
I: 3 = game.ai.combat.v0.ArchetypeComponent (85952)
I: 4 = game.ai.combat.v0.InterestedInItemsComponent (16)
I: 5 = game.ai.swarm.v0.MemberComponent (272)
I: 6 = game.ai.swarm.v0.Group (800)
I: 7 = game.ai.swarm.v0.GroupsComponent (16)
I: 8 = game.approval.v0.Ratings (480)
I: 9 = game.v0.StateComponent (56)
I: 10 = game.attitude.v0.AttitudeEntry (48)
I: 11 = game.attitude.v0.AttitudesToPlayersComponent (4112)
I: 12 = game.avatar.v0.AvatarComponent (16)
I: 13 = game.background.v0.BackgroundGoals (96)
I: 14 = game.background.v0.GoalRecord (224)
I: 15 = game.background.v0.GoalsComponent (16)
I: 16 = game.body_type.v0.BodyTypeComponent (4112)
I: 17 = game.breadcrumb.v0.BreadcrumbComponent (32)
I: 18 = game.calendar.v0.StartingDateComponent (8)
I: 19 = game.calendar.v0.DaysPassedComponent (4)
I: 20 = game.camp.v0.TotalSuppliesComponent (4)
I: 21 = game.camp.v0.QualityComponent (8)
I: 22 = game.camp.v0.SupplyComponent (192)
I: 23 = game.camp.v1.EndTheDayStateComponent (16)
I: 24 = game.camp.v1.ChestComponent (48)
I: 25 = game.camp.v1.PresenceComponent (4)
I: 26 = game.camp.v0.SettingsComponent (12)
I: 27 = game.camp.v0.TriggerComponent (24)
I: 28 = game.capabilities.v0.CanBeLootedComponent (2056)
I: 29 = game.capabilities.v0.CanDoActionsComponent (2056)
I: 30 = game.capabilities.v0.CanDoRestComponent (64)
I: 31 = game.capabilities.v2.CanInteractComponent (4112)
I: 32 = game.capabilities.v0.CanModifyHealthComponent (2056)
I: 33 = game.capabilities.v1.CanMoveComponent (6168)
I: 34 = game.capabilities.v0.CanSenseComponent (2088)
I: 35 = game.capabilities.v0.CanSpeakComponent (2056)
I: 36 = game.capabilities.v0.CanTravelComponent (6168)
I: 37 = game.capabilities.v1.CanTriggerRandomCastsComponent (2056)
I: 38 = game.capabilities.v1.FleeCapabilityComponent (4112)
I: 39 = game.character.v0.CharacterComponent (264)
I: 40 = game.character.v0.EquipmentVisualComponent (80)
I: 41 = game.character_creation.v0.BackgroundComponent (96)
I: 42 = game.character_creation.v1.AppearanceMaterialSetting (3024)
I: 43 = game.character_creation.v3.AppearanceComponent (784)
I: 44 = game.character_creation.v3.LevelUpComponentData (2688)
I: 45 = game.character_creation.v0.LevelUpComponentAbilities (448)
I: 46 = game.character_creation.v2.LevelUpComponentSelectors (3136)
I: 47 = game.character_creation.v1.SelectorMeta (2880)
I: 48 = game.character_creation.v2.BaseSelector (1920)
I: 49 = game.character_creation.v1.AbilityAddSlot (192)
I: 50 = game.character_creation.v2.AbilityBonusSelector (672)
I: 51 = game.character_creation.v1.SkillAddSlot (368)
I: 52 = game.character_creation.v2.SkillSelector (720)
I: 53 = game.character_creation.v1.StringViewAddSlot (896)
I: 54 = game.character_creation.v2.SpellSelector (1344)
I: 55 = game.character_creation.v2.PassiveSelector (112)
I: 56 = game.character_creation.v2.SkillExpertiseSelector (192)
I: 57 = game.character_creation.v3.LevelUpComponent (96)
I: 58 = game.character_creation.v0.StateComponent (8)
I: 59 = game.character_creation.v1.CharacterCreationStatsComponent (528)
I: 60 = game.character_creation.v0.OriginComponent (96)
I: 61 = game.character_creation.v0.VoiceComponent (96)
I: 62 = game.character_creation.v0.AppearanceVisualTagComponent (32)
I: 63 = game.character_creation.v0.GodComponent (96)
I: 64 = game.character_creation.v0.IsCustomComponent (8)
I: 65 = game.combat.v0.ParticipantComponent (64368)
I: 66 = game.combat.v0.CanStartCombatComponent (232)
I: 67 = game.spell.v0.SpellSource (125568)
I: 68 = game.spell.v0.MetaId (125568)
I: 69 = game.spell.v0.SpellId (72840)
I: 70 = game.concentration.v0.ConcentrationComponent (6168)
I: 71 = game.darkness.v1.DarknessComponent (8224)
I: 72 = game.darkness.v0.DarknessActiveComponent (8)
I: 73 = game.death.v0.DeadByDefaultComponent (8)
I: 74 = game.death.v2.DeathData (240)
I: 75 = game.death.v2.DeathComponent (16)
I: 76 = game.death.v1.StateComponent (8)
I: 77 = game.death.v1.DelayDeathReasons (1032)
I: 78 = game.death.v1.DelayDeathCauseComponent (4112)
I: 79 = game.v1.DetachedComponent (8)
I: 80 = game.dialog.v0.ADRateLimitingHistoryComponent (16)
I: 81 = game.dialog.v0.StateComponent (264)
I: 82 = game.display_names.v0.DisplayNameTS (78112)
I: 83 = core.v0.TranslatedString (122816)
I: 84 = game.display_names.v0.DisplayTitleTS (800)
I: 85 = game.display_names.v0.Component (42976)
I: 86 = game.dual_wielding.v0.DualWieldingComponent (1800)
I: 87 = game.experience.v0.AvailableLevelComponent (1028)
I: 88 = game.experience.v0.ExperienceComponent (48)
I: 89 = game.experience.v0.ExperienceGaveOutComponent (1028)
I: 90 = game.game_timer.v1.GameTimerComponent (720)
I: 91 = game.god.v0.GodComponent (240)
I: 92 = game.god.v0.TagComponent (96)
I: 93 = game.gravity.v0.GravityDisabledComponent (72)
I: 94 = game.gravity.v0.GravityDisabledUntilMovedComponent (400)
I: 95 = game.hotbar.v4.Container (160)
I: 96 = game.hotbar.v4.Bar (7840)
I: 97 = game.hotbar.v1.Slot (18576)
I: 98 = game.hotbar.v4.Component (560)
I: 99 = game.hotbar.v0.OrderComponent (16)
I: 100 = game.icon.v0.CustomIconComponent (96)
I: 101 = game.icons.v0.Icon (32760)
I: 102 = game.icons.v0.Component (32232)
I: 103 = game.identity.v0.IdentityComponent (80)
I: 104 = game.identity.v0.OriginalIdentityComponent (48)
I: 105 = game.identity.v0.StateComponent (80)
I: 106 = game.improvisedweapon.v0.CanBeWieldedComponent (336)
I: 107 = game.interrupt.v0.PreferencesComponent (7616)
I: 108 = game.inventory.v0.CanBeInComponent (1032)
I: 109 = game.inventory.v0.ContainerSlotData (15392)
I: 110 = game.inventory.v1.ContainerComponent (18048)
I: 111 = game.inventory.v3.Type (4512)
I: 112 = game.inventory.v4.DataComponent (9024)
I: 113 = game.inventory.v1.IsOwnedComponent (4512)
I: 114 = game.inventory.v0.MemberData (15664)
I: 115 = game.inventory.v0.MemberComponent (7832)
I: 116 = game.inventory.v0.OwnerComponent (7368)
I: 117 = game.inventory.v0.StackEntry (1064)
I: 118 = game.inventory.v0.Stack (3488)
I: 119 = game.inventory.v0.NewStackComponent (872)
I: 120 = game.inventory.v0.StackMemberComponent (1008)
I: 121 = game.inventory.v0.WieldedComponent (832)
I: 122 = game.inventory.v0.WieldingComponent (3256)
I: 123 = game.inventory.v0.CharacterHasGeneratedTradeTreasureComponent (4)
I: 124 = game.inventory.v1.ContainerDataComponent (4512)
I: 125 = game.inventory.v0.ItemHasGeneratedTreasureComponent (44)
I: 126 = game.inventory.v0.ShapeshiftEquipmentHistoryComponent (96)
I: 127 = game.inventory.v0.InventoryPropertyIsDroppedOnDeathComponent (5160)
I: 128 = game.inventory.v0.InventoryPropertyIsTradableComponent (5160)
I: 129 = game.invisibility.v3.InvisibilityComponent (480)
I: 130 = game.item.v0.HasMovedComponent (22)
I: 131 = game.item.v0.HasOpenedComponent (43)
I: 132 = game.item.v0.ItemComponent (1086)
I: 133 = game.item.v0.CanMoveComponent (1028)
I: 134 = game.item.v0.InteractionDisabledComponent (17)
I: 135 = game.item.v0.IsStoryItemComponent (88)
I: 136 = game.jumpfollow.v0.JumpFollowComponent (168)
I: 137 = game.v0.InventoryItemDataPopulatedComponent (1148)
I: 138 = game.lock.v0.KeyComponent (320)
I: 139 = game.lock.v0.V1LockComponent (96)
I: 140 = game.lootvalidation.v3.LootComponent (514)
I: 141 = game.v0.LevelIsOwnerComponent (1718)
I: 142 = game.v0.SavegameComponent (1736)
I: 143 = game.v0.SaveWithComponent (4632)
I: 144 = game.v0.IsGlobalComponent (2291)
I: 145 = game.v0.OffStageComponent (20)
I: 146 = game.v0.OwnedAsLootComponent (881)
I: 147 = game.v0.OwneeCurrentComponent (8688)
I: 148 = game.v1.OwneeHistoryComponent (34752)
I: 149 = game.v0.IsCurrentOwnerComponent (2768)
I: 150 = game.v0.IsLatestOwnerComponent (2768)
I: 151 = game.v1.IsPreviousLatestOwnerComponent (176)
I: 152 = game.v1.IsPreviousOwnerComponent (192)
I: 153 = game.v0.IsOriginalOwnerComponent (2800)
I: 154 = game.v0.OwneeRequestComponent (34752)
I: 155 = game.party.v0.CompositionComponent (32)
I: 156 = game.party.v0.MemberComponent (192)
I: 157 = game.party.v0.PortalsComponent (16)
I: 158 = game.party.v0.RecipeData (192)
I: 159 = game.party.v1.RecipesComponent (16)
I: 160 = game.party.v0.ViewComponent (8)
I: 161 = game.party.v0.WaypointsComponent (16)
I: 162 = game.party.v0.UserGroupSnapshot (16)
I: 163 = game.party.v0.UserSnapshotComponent (32)
I: 164 = game.passives.v0.PersistentDataComponent (10744)
I: 165 = game.passives.v0.ToggledPassivesComponent (42976)
I: 166 = game.passives.v1.UsageCountComponent (32)
I: 167 = game.passives.v0.ScriptPassivesComponent (16)
I: 168 = game.pickpocket.v0.PickpocketComponent (4112)
I: 169 = game.pickpocket.v0.InventoryPropertyCanBePickpocketedComponent (5160)
I: 170 = game.v0.PlayerComponent (4)
I: 171 = game.v0.ClientControlComponent (4)
I: 172 = game.progression.v3.LevelUpComponent (4112)
I: 173 = game.race.v0.RaceComponent (4112)
I: 174 = game.recruit.v0.RecruiterComponent (16)
I: 175 = game.recruit.v0.RecruitedByComponent (40)
I: 176 = game.relation.v0.FactionRelation (21840)
I: 177 = game.relation.v1.RelationComponent (96)
I: 178 = game.relation.v0.FactionComponent (53720)
I: 179 = game.repose.v2.StateComponent (336)
I: 180 = game.repose.v0.UsedEntitiesToCleanSingletonComponent (16)
I: 181 = game.roll.stream.v1.StreamsComponent (32)
I: 182 = game.safe_position.v0.SafePositionComponent (4112)
I: 183 = game.shapeshift.v0.ChangeInt (336)
I: 184 = game.shapeshift.v1.SharedShapeshiftComponent (96696)
I: 185 = game.shapeshift.v0.HealthReservationComponent (42976)
I: 186 = game.shapeshift.v5.State (3072)
I: 187 = game.shapeshift.v5.ServerShapeshiftComponent (21488)
I: 188 = game.sight.v0.DataComponent (21488)
I: 189 = game.sight.v0.EntityViewshedComponent (3392)
I: 190 = game.sight.v0.ViewshedParticipantComponent (6592)
I: 191 = game.spell.v1.SpellMeta (64)
I: 192 = game.spell.v1.AddedSpellsComponent (4112)
I: 193 = game.spell.v2.SpellData (171216)
I: 194 = game.spell.v1.CastRequirements (171216)
I: 195 = game.spell.v2.SpellBookComponent (4896)
I: 196 = game.spell.v1.CooldownData (336)
I: 197 = game.spell.v1.SpellBookCooldowns (4112)
I: 198 = game.spell.v0.SpellBookPrepares (20560)
I: 199 = game.spell.v0.CCPrepareSpellComponent (96)
I: 200 = game.spell.v0.LearnedSpells (8224)
I: 201 = game.spell.v0.PlayerPrepareSpellComponent (144)
I: 202 = game.spell.v0.ScriptedExplosionComponent (112)
I: 203 = game.spell.v0.OnDamageSpell (224)
I: 204 = game.spell.v0.OnDamageSpellsComponent (112)
I: 205 = game.spell_cast.v0.SpellData (336)
I: 206 = game.spell_cast.v0.DataCacheSingletonComponent (16)
I: 207 = game.splatter.v0.StateComponent (7200)
I: 208 = game.stats.v0.ClassesComponent (4112)
I: 209 = game.stats.v1.DifficultyCheckComponent (2056)
I: 210 = game.stats.v0.HealthComponent (14880)
I: 211 = game.stats.v0.LevelComponent (1028)
I: 212 = game.stats.v0.AreaLevelComponent (12)
I: 213 = game.stats.v3.StatsComponent (9252)
I: 214 = game.stats.v0.UseComponent (5980)
I: 215 = game.status.v0.IncapacitatedComponent (168)
I: 216 = game.status.visual.v0.DisabledComponent (16)
I: 217 = game.tadpole_tree.v0.TadpoledComponent (8)
I: 218 = game.tadpole_tree.v1.TreeStateComponent (40)
I: 219 = game.tags.v0.VoiceComponent (16)
I: 220 = game.tags.v0.AnubisComponent (27488)
I: 221 = game.tags.v0.DialogComponent (27488)
I: 222 = game.tags.v0.OsirisComponent (27488)
I: 223 = game.tags.v0.RaceComponent (4112)
I: 224 = game.tags.v0.TemplateComponent (27488)
I: 225 = game.templates.v0.TemplateComponent (32232)
I: 226 = game.through.v0.CanSeeThroughComponent (1342)
I: 227 = game.through.v0.CanShootThroughComponent (1002)
I: 228 = game.through.v0.ShootThroughTypeComponent (10744)
I: 229 = game.through.v0.CanWalkThroughComponent (1024)
I: 230 = game.timeline.v1.ActorVisualDataComponent (64)
I: 231 = game.triggers.v0.ContainerComponent (128)
I: 232 = game.triggers.v0.InInsideOfTriggerComponent (21488)
I: 233 = game.triggers.v0.ActiveMusicVolumeComponent (80)
I: 234 = game.triggers.v1.CachedLeaveEventsComponent (21488)
I: 235 = game.triggers.v0.RegisteredForTriggersComponent (4112)
I: 236 = game.triggers.v0.RegistrationSettingsComponent (8)
I: 237 = game.turn_based.v0.ParticipantComponent (21488)
I: 238 = game.turn_based.v3.TurnBasedComponent (75208)
I: 239 = tutorial.v0.ProfileEventDataComponent (32)
I: 240 = game.unsheath.v8.StateComponent (10280)
I: 241 = game.unsheath.v0.DefaultComponent (368)
I: 242 = game.unsheath.v0.ScriptOverrideComponent (48)
I: 243 = game.visual.v4.GameObjectVisualComponent (247112)
I: 244 = game.v0.WeaponSetComponent (2056)
I: 245 = game.v0.EState (8)
I: 246 = game.body_type.v0.EBodyType (16)
I: 247 = game.attitude.v0.EIdentityState (8)
I: 248 = game.camp.v1.EEndTheDayState (8)
I: 249 = game.capabilities.v0.ELootableCapabilities (16)
I: 250 = game.capabilities.v0.EActionCapabilities (16)
I: 251 = game.capabilities.v0.ERestCapabilities (8)
I: 252 = game.capabilities.v1.EInteractionCapabilities (48)
I: 253 = game.capabilities.v1.EInteractionError (16)
I: 254 = game.capabilities.v0.EModifyHealthCapabilities (8)
I: 255 = game.capabilities.v1.EMovementCapabilities (56)
I: 256 = game.capabilities.v0.EMovementError (8)
I: 257 = game.capabilities.v0.EPathMovementSpeed (8)
I: 258 = game.capabilities.v0.EAwarenessCapabilities (56)
I: 259 = game.capabilities.v0.ESpeakingCapabilities (32)
I: 260 = game.capabilities.v0.ETravelCapabilities (8)
I: 261 = game.capabilities.v0.ETravelError (8)
I: 262 = game.capabilities.v0.EGatherAtCampError (16)
I: 263 = game.capabilities.v1.ERandomCastError (8)
I: 264 = game.capabilities.v0.EFleeBlock (40)
I: 265 = game.character.v0.ECharacterStowedOption (8)
I: 266 = game.character_creation.v1.ESelectorOwnerType (32)
I: 267 = game.character_creation.v1.EAbility (48)
I: 268 = game.character_creation.v1.ESkill (96)
I: 269 = game.character_creation.v1.EBodyShape (8)
I: 270 = game.identity.v0.EIdentity (24)
I: 271 = game.combat.v0.ECombatParticipantComponentFlags (80)
I: 272 = game.spell.v0.ESourceType (104)
I: 273 = game.darkness.v1.EDarknessActiveSource (24)
I: 274 = game.darkness.v1.EObscuredState (24)
I: 275 = game.death.v1.EDeathType (16)
I: 276 = game.death.v2.TCauseType (8)
I: 277 = game.v0.DetachOrigin (8)
I: 278 = game.hotbar.v2.EHotBarType (72)
I: 279 = game.hotbar.v0.EHotBarControllerType (24)
I: 280 = game.identity.v0.EIdentityState (8)
I: 281 = game.interrupt.v0.EInteractionType (24)
I: 282 = game.inventory.v0.EIsTradableType (16)
I: 283 = game.invisibility.v3.EInvisibilitySourceType (8)
I: 284 = game.relation.v0.ERelation (32)
I: 285 = game.shapeshift.v0.EChangeType (8)
I: 286 = game.shapeshift.v0.EItemTooltipChange (8)
I: 287 = game.shapeshift.v0.EIdentityState (24)
I: 288 = game.shapeshift.v0.ECharacterFootStepsType (40)
I: 289 = game.shapeshift.v0.EBodyType (32)
I: 290 = game.shapeshift.v0.EActionCapabilities (16)
I: 291 = game.shapeshift.v0.EInteractionCapabilities (8)
I: 292 = game.shapeshift.v0.EAwarenessCapabilities (8)
I: 293 = game.shapeshift.v0.ESpeakingCapabilities (16)
I: 294 = game.templates.v0.ETemplateHandleType (40)
I: 295 = game.shapeshift.v0.EArmorType (32)
I: 296 = game.shapeshift.v0.EAbility (64)
I: 297 = game.size.v0.EObjectSize (24)
I: 298 = game.spell.v0.ELearningStrategy (8)
I: 299 = game.spell.v0.EPreparationStrategy (8)
I: 300 = game.spell.v0.EAbility (48)
I: 301 = game.spell.v1.ECooldownType (56)
I: 302 = game.spell.v1.ESpellRequirementType (88)
I: 303 = game.spell.v0.ESpellSchool (8)
I: 304 = game.damage.v0.EDamageType (8)
I: 305 = game.status.v0.EIncapacitationReason (16)
I: 306 = game.tadpole_tree.v1.ETadpoleTreeState (8)
I: 307 = game.unsheath.v7.EPriority (40)
I: 308 = game.unsheath.v0.EState (32)
I: 309 = game.unsheath.v0.ECause (16)
I: 310 = game.v0.EWeaponSet (0)
@LennardF1989 I can't provide anything meaningful here and it's unlikely the best place to ask but can you recommend any resources to learn this type of RE or even some keywords I might lookup to find the right type of content?
RE-ing files is quite hard to explain, haha:P Get a dump of a save-game (using LSLib), extract the NewAge portion, base64 decode it, then stare at hex data in a Hex Editor (I'm using ImHex now for the template feature instead of the commercial 010 Editor) and try to make sense of what you are seeing.
Right now I found the name table for the components (see above), I found the WebP files for the avatars. I think I found something that resembles your base stats (STR/DEX/etc). I have a feeling there is a bit of LSV format in this as well (see LSLib's LSFReader.cs on how that is generally read) when it comes to node-structures (it's a returning pattern in their files, so why not in this one too).
This is my ImHex pattern so far (it's pretty much a port of the above 010 one):
#include <std/io.pat>
struct Block0Entry<auto offset> {
u64 nameOffset;
u64 nameSize;
u8 unknown1[8];
u32 flag1;
u32 flag2;
u32 flag3;
u32 flag4;
u64 componentOffset;
char name[nameSize] @ offset + nameOffset;
};
struct Block0Header {
u64 relativeOffset;
u64 unknown1;
u32 nameSize;
u16 totalNameEntries;
u8 unknown2[10];
u64 absoluteOffset = $ + relativeOffset;
Block0Entry<absoluteOffset> nameEntries[totalNameEntries] @ absoluteOffset + nameSize;
};
struct Block1Header {
u64 currentOffset = $;
u8 unknown1[8];
u64 relativeOffset; //NOTE: Relative to start of this block header?
u64 unknown2;
u64 unknownSize1;
u64 unknown3;
//u8 unknown4[unknownSize1] @ currentOffset + relativeOffset;
};
struct LSMF {
char magicHeader[4];
u8 unknown1[12];
Block0Header block0Header;
Block1Header block1Header;
};
struct CharacterStats {
//char name[while(std::mem::read_unsigned($, 1) != 0x00)];
//std::mem::read_unsigned($, 1);
char name[8];
u32 stat1;
u32 stat2;
u32 stat3;
u32 stat4;
u32 stat5;
u32 stat6;
u32 stat7;
};
LSMF lsmf @ 0x0;
CharacterStats stats @ 0x002F96FC;
std::print("Name entries");
for(u32 i = 0, i < lsmf.block0Header.totalNameEntries, i = i + 1) {
str name = lsmf.block0Header.nameEntries[i].name;
u32 size = 0;
if(i < lsmf.block0Header.totalNameEntries - 1) {
size = lsmf.block0Header.nameEntries[i + 1].componentOffset - lsmf.block0Header.nameEntries[i].componentOffset;
}
std::print("{:3d} = {} ({})", i, name, size);
}
For simplicity, I've attached the NewAge file I'm using (saves you from trying to get one - extract it first): NewAge.zip
Managed to map another whole range of strings, found it by accident as I was scanning for repeating patterns.
Offset into my NewAge file.
struct Test {
u64 stringOffset1;
u32 stringSize1;
u32 unknown1;
u64 unknown2;
u64 unknown3;
u64 stringOffset2;
u32 stringSize2;
u8 unknown[140];
char string1[stringSize1] @ stringOffset1 + 48;
char string2[stringSize2] @ stringOffset2 + 48;
};
Test test[1343] @ 0x00261360;
Lot's of duplicate key/valye pairs, so the unknown bits probably have something interesting in it.
I: Test entries
I: f65becd6-5cd7-4c88-b85e-6dd06b60f7b8 = dc5589d3-5f3b-0ac4-ef9d-88c34dd85f9c-EQP_Unarmed_(Icon_Raphael_Human)
I: 475200ee-cc3c-4dbe-84b1-1820c02ea26a = 6b49f80c-3ce2-cfe8-f569-5202f8f09f23-DEN_TieflingLeader_(Icon_Tiefling_Male)
I: 27fa0802-fa38-4eea-9c03-496f2e022259 = 6f810419-6a19-e9eb-e8b7-690910c15ca8-GLO_Gith_Captain_(Icon_Githyanki_Female)
I: 5dd3bb4a-97fa-48b6-9489-5cd577d217f2 = 8a15f6ea-31c5-aa95-89dd-0baa471c08ce-EQP_HeavyCrossbow_StuddedLeather_Gith_(Icon_Githyanki_Male)
...
Things are starting to fall in place now that I know that every offset is probably minus/plus 48. I found a way to map a large portion of strings (also "referenced strings"). These include the character names in your party, your profile id, etc.
struct Test2SubString {
u64 stringOffset;
u32 stringSize;
u8 unknown1[4];
char string[stringSize] @ stringOffset + 48;
};
struct Test2 {
u64 stringOffset1;
u32 stringSize1; //Type?
u8 unknown1[4];
u64 stringOffset2;
u32 stringSize2;
u8 unknown2[4];
str name = "" [[export]];
if(stringOffset1 == 18446744073709551615) {
char directString[stringSize2] @ stringOffset2 + 48;
name = directString;
}
else {
Test2SubString subString @ stringOffset1 + 48;
name = subString.string;
}
};
Test2 test2[133] @ 0x000CCD20;
Output:
I: Test 2 entries
...
I: 43 = h29b894bcg6d9cg4d63ga57bg52cf4f8a28bf
I: 44 = h495ac5cag0532g4324ga544ge1a35064a12e
I: 45 = h4ebf5b0cgd1f9g48a4g856cg9c78aee13965
I: 46 = hdb8fc8dfgf320g4f5fga958g3f28960fb92e
I: 47 = hd4c23155ge911g4ffag9ba5gd0cb37841d6f
I: 48 = Wyll
...
I: 938 = h3d7a0345g2af5g4bb9gb592gb72fb43588b8
I: 939 = ResStr_118253367
...
Before I head to bed, last bit of info I have on the first bit of the file (after the huge chunk of unknown data):
struct Test5Header {
u64 startOffset;
u64 endOffset;
u8 unknownBody[endOffset - startOffset] @ startOffset + 48;
};
struct Test7Block {
Test5Header lsmf_top1[257];
Test5Header lsmf_top2_1;
Test2SubString lsmf_top2_2[5372];
Test5Header lsmf_top3_1;
Test2SubString lsmf_top3_2[18];
Test5Header lsmf_top4_1;
Test2SubString lsmf_top4_2[1];
Test5Header lsmf_top5_1;
Test2SubString lsmf_top5_2[1];
Test5Header lsmf_top6_1;
Test2SubString lsmf_top6_2[1];
};
Test7Block lsmf_top @ 0x00043558;
It's WIP as I'm still looking for the logic how it decides how many substrings follow after a header. I think Test5Header and Test2SubStrings should be combined based on the values it reads, and instead of all kinds of small chunks, it's actually a list of 6000-ish structures. There is a lot of FF FF FF FF, so they could also be separate arrays that have a particular size pre-reserved. Who knows! We'll find it :)
EDIT: Also, I found the block that describes where to find the WebP files, see offset 0x00110868. It's a list of start/end offsets.
Have you tried modifying some of this and loading it back into your game to see if it sticks?
I've meticulously gone through every LSV/LSX attempting to remove a custom character from my party, recompile the save, and they're still there as if nothing changed... So I assume it's all thanks to NewAge keeping the "state" of things.
Would be interesting to take NewAge from a new save with 1 player, then another after adding a second custom player and diff the two strings...
I haven't yet, this is all by just analyzing. Changing stuff other than some stats without touching the integrity of the file is going to be hard. Most of the format will have to be digested before we can even go as far as re-saving it. There is so much stuff pointing at other stuff. Eg. Say you want to rename your character to a name with more characters than you originally had, everything that points at this particular string will need to know the new size. And everything that shifts will need to have their reference position updated.
Some context for NewAge:
Even though D:OS2 had an entity-component system, most of the logic was historically packed into two giga-components, esv::Character and esv::Item (which had their own LSF nodes).
When upgrading the engine for BG3, these large components were split up many small components, each with their own storage.
Because the serialization for so many components into LSF was very slow (looking up each field by name, lots of extra metadata kept, etc.), a new serialization method was introduced, that stores component data as-is, without any metadata tagging, which is NewAge. It is essentially a binary that contains a list of all entity components for all entities, and a serialized representation of their own internal data. Sadly there is no way to programatically interpret what the serialized data within each component means, you either have to map the serializers in bg3.exe or try to guess the meaning of various bytes.
Good to know! We can probably get away with not understanding every byte to modify some portions of it. I've manually "deserialized" some of the components now. I think people will be mostly interested (at least I am), to modify your character model, stats and maybe some other bits, that (probably) doesn't require reserializing the whole file.
I have almost all bytes referenced by something now (I've pretty much figured out how the header finds the components, how the components find their data). Out of 4MB, only 1MB is left untouched (but I also know why).
Other than that, I like the challenge of solving these kinds of things, whether or not it will lead to something useful, haha.
@LennardF1989 just wondering if you've had any luck with this! I'm also interested in parsing out character stats (mostly to recreate characters between saves but also out of interest in seeing what can be modified), but haven't had any luck in trying my own hand at this.
I have parked it for a moment to actually stop tinkering and enjoy the game for a bit, haha. As Norbyte mentioned, being able to modify data is going to be hard, reading it should be possible to a certain extend. I have made a bunch of different saves with slight alterations between the characters, and the NewAge data is only slightly different. Those should be the parts that the determine how the character looks. I will resume and share findings somewhere this week :)
Haha fair enough! When you say you made a bunch of different saves with slight alterations between the characters, did you just write down or roughly remember what you did for each character, or do you know if there's a way at all to see what options were chosen at character creation?
Haha fair enough! When you say you made a bunch of different saves with slight alterations between the characters, did you just write down or roughly remember what you did for each character, or do you know if there's a way at all to see what options were chosen at character creation?
I remembered what I did and made subtle changes like eye color, hair style, nothing too fancy.
I have not found anything conclusive yet to say "Oh, this means gold blond 5, and this is deep blue, or anything yet." But I'll get there :)
Someone @ Nexus Mods managed to fiddle with similar appearance items, but I don't believe they are doing anything with NewAge: https://www.nexusmods.com/baldursgate3/mods/899
Did NewAge change at all after the hotfixes Larian made to allow save files to be larger?
This is by starting the character creation again. I've played with that, but evident by the large description in that mod, but it's far from ideal.
The NewAge format doesn't change, but the contents can change. There are components in there that describe stuff, but they are versioned. So a component that's v1 one now, can have a new layout in v2.
I haven't tested my theory yet, but some of these components Norbyte has already mapped out in his bg3se (see https://github.com/Norbyte/bg3se/blob/main/BG3Extender/GameDefinitions/PropertyMaps/Components.inl), and I'm pretty sure that can translate to NewAge in a lot of cases, to at least get an idea of what is where and what it means. My initial goal is to be able to modify some basic character appearance things, like eye color or hair color, then go from there.
This is by starting the character creation again. I've played with that, but evident by the large description in that mod, but it's far from ideal.
The NewAge format doesn't change, but the contents can change. There are components in there that describe stuff, but they are versioned. So a component that's v1 one now, can have a new layout in v2.
I haven't tested my theory yet, but some of these components Norbyte has already mapped out in his bg3se (see
https://github.com/Norbyte/bg3se/blob/main/BG3Extender/GameDefinitions/PropertyMaps/Components.inl), and I'm pretty sure that can translate to NewAge in a lot of cases, to at least get an idea of what is where and what it means. My initial goal is to be able to modify some basic character appearance things, like eye color or hair color, then go from there.
How would you go about approaching modifying these components? My first thought would be the ComponentHandle or Entity classes?
just to provide some data to compare, here's a NewAge that I pulled from my save with my partner. NewAge.mmetully.zip
Two Custom Characters Host: Shanaila Guest: Nasmira
handful of mods.
I'm not sure that our progress is going to impact this data, but our game is just progressed past the tutorial
u32 size = 0; if(i < lsmf.block0Header.totalNameEntries - 1) { size = lsmf.block0Header.nameEntries[i + 1].componentOffset - lsmf.block0Header.nameEntries[i].componentOffset; }
Is this correct @LennardF1989 ? Just found it strange that core.v0.EntityId reports such a huge size.
I: Name entries
I: 0 = core.v0.Level (32)
I: 1 = core.v0.EntityId (146512)
I'm stuck trying to fin the start/end of each component. Tried to use componentOffset, but it's not close to anything that I can see.
It is correct, it's a list of GUIDs - most likely with all items spawned into the world.
I'm slowly picking up reversing this again and will share my results over this week.
I modified the for-loop a bit. Seems to get the correct offset for each component now.
for(u32 i = 0, i < lsmf.block0Header.totalNameEntries, i = i + 1) {
str name = lsmf.block0Header.nameEntries[i].name;
u32 size = 0;
if(i < lsmf.block0Header.totalNameEntries - 1) {
size = lsmf.block0Header.nameEntries[i + 1].componentOffset - lsmf.block0Header.nameEntries[i].componentOffset;
}
std::print("{:3d} = {} ({}) 0x{:X}", i, name, size, lsmf.block0Header.nameEntries[i].componentOffset + 48);
}
I: Name entries
I: 0 = core.v0.Level (32) 0x38
I: 1 = core.v0.EntityId (146512) 0x58
I: 2 = game.action_resources.v1.Component (4080) 0x23CA8
I also noticed that core.v0.Level overlaps with most of Block1Header.
I also noticed that core.v0.Level overlaps with most of Block1Header.
0x38 (offset 0) = 56
56 + 32 (size) = 88
88 = 0x58 (offset 1)
It's correct :)
Block1Header is near the bottom of the file, so I think you misread.
I started with your template above. For some reason the Block1Header doesn't offset below the Block0Entries.
I'm waiting/working on a fix (crash-fix) for a new feature in ImHex to use dynamic groups in the pattern editor to get something like this:
So far I've reversed (I think) the most important ones, and while I have not tried (yet, but I will), I'm confident I can tweak some things in the saves to modify appearance, voice, origin, etc.
I'm waiting/working on a fix (crash-fix) for a new feature in ImHex to use dynamic groups in the pattern editor to get something like this:
So far I've reversed (I think) the most important ones, and while I have not tried (yet, but I will), I'm confident I can tweak some things in the saves to modify appearance, voice, origin, etc.
@LennardF1989 Would you be willing to share the pattern that you have as of now? I found this issue after losing some days trying to figure out the "NewAge" structure and it seems you are way ahead of me.
I'm not quite as far as Lennard, but here goes:
The primary header was more or less covered in the initial post already, but since I've renamed quite a few of them, once again in full.
Apart from naming, the only thing that I added was stuff in the ComponentInfo (formerly Block0Entry) struct.
#include <std/io.pat>
#include <std/mem.pat>
#include <std/limits.pat>
struct ComponentInfo {
u64 nameOffset;
u64 nameSize;
u64 unknown; // component signature? seems to stay the same
u32 elementSize;
u32 version;
u64 elementCount;
u64 componentOffset;
u64 size = elementCount * elementSize;
char name[nameSize] @ parent.infoOffset + nameOffset + 48;
};
struct LSMF {
char magicHeader[4];
u8 version[4];
u64 hash;
u64 infoOffset;
u64 indexSize;
u32 nameSize;
u16 componentCount;
ComponentInfo components[componentCount] @ infoOffset + nameSize + 48;
u16 unknown2;
u64 unknown3;
u64 unknown4;
};
LSMF lsmf @ 0x0;
As such, the loop becomes
for(u32 i = 0, i < lsmf.componentCount, i = i + 1) {
str name = lsmf.components[i].name;
u32 elementSize = lsmf.components[i].elementSize;
u64 elementCount = lsmf.components[i].elementCount;
u64 offset = componentOffset(i);
std::print("{:3d} = {} ({} * {}) 0x{:X}", i, name, elementCount, elementSize, offset);
}
Two utility functions used for component alignment.
fn componentOffset(u16 i) {
return lsmf.components[i].componentOffset + 48;
};
fn elementCount(u16 i) {
return lsmf.components[i].elementCount;
};
The second important thing as far as structure goes is the game.v0.Level component. To my current understanding it is what provides a mapping from component entries to their owner entities.
The first two numbers bound a range of ids, the second an array of structs in the 'Heap' (The section of the file past the named components).
These are serialized somewhat strangely, there are more of them than components, but most of them are invalid (hence the check for the component being u64_max) to the point that not every component has an associated owner list.
Based on the respective value ranges ranges, I interpret these lists as being read in parallel to their component (They always have the same length).
Then the owner of the i-th entry of a component would be level.entityIndex[ownerList.owners[i]].
struct EntityId {
u8 id[16];
};
struct OwnerList {
u64 start;
u64 end;
u64 component;
u64 elementCount;
if (start != std::limits::u64_max()) {
u32 owners[elementCount] @ start + 48;
}
};
struct Level {
u64 entityStart;
u64 entityEnd;
u64 ownersStart;
u64 ownersEnd;
u64 entityCount = (entityEnd - entityStart)/16;
u64 ownerCount = (ownersEnd - ownersStart)/32;
EntityId entityIndex[entityCount] @ entityStart + 48;
OwnerList ownerLists[ownerCount] @ ownersStart + 48;
};
Level level @ componentOffset(0);
For convenience I've also made a lookup for the OwnerList of each component, though there's the obvious weakness that you have to make sure their sizes match in case you get 0. I simply check manually beforehand, but you could obviously fill empty entries with a value that'd error if used.
std::mem::Section lookupSection = std::mem::create_section("Lookup Section");
u16 componentLookup[lsmf.componentCount] @ 0x0 in lookupSection;
for (i = 0, i < level.ownerCount, i = i + 1) {
if (level.ownerLists[i].component != std::limits::u64_max()) {
componentLookup[level.ownerLists[i].component] = i;
}
}
This roughly divides the components into 'top level' ones that directly have owners for their entries and the rest, which have their data referred to by others. I'm not entirely sure what the purpose of the latter (as opposed to just putting them on heap) is yet, since everything I have come across for now (apart from the owner lists) retrieves its data by address rather than index.
On the other hand, something I noticed while trying to map the party characters to their names is that the name components introduce seemingly unused entries with each 'layer', i.e. from display_names.Component trough display_names.DisplayNameTS to TranslatedString, which mostly consist of references to the next, do not cover all entries, and searches for the missed addresses did not yield results.
The character names were one of the consistently affected, the only way I have found for programmatically retrieving them for now is to go through character_creation.CharacterCreationStatsComponent, whose entries each hold a reference to the second occurrence of the custom names within the file.
Edit: The ordering of the components in the index seems to be mutable, so identification by index (like done by the utility functions above) might break between saves and mustn't be relied on for fully automatic applications.
The old link is dead due to a refactor, but @LennardF1989 's idea to cross compare against the findings of the script extender has proven extremely helpful.
The appearance component for example seems to be a serialization of CharacterCreationAppearanceComponent. (Visuals.h#117 at the time of writing)
Along with the material settings struct from the same file, the save entry would likely be read like this:
struct MaterialData {
Guid material;
Guid color;
float colorIntensity;
float metallicTint;
float glossyTint;
u32 unknown;
};
struct AppearanceChunk {
u64 visualsStart;
u64 visualsEnd;
u64 materialsStart;
u64 materialsEnd;
Guid skinColor;
u64 choicesStart;
u64 choicesEnd;
Guid visuals[(visualsEnd - visualsStart) / 16] @ visualsStart + 48;
MaterialData materials[(materialsEnd - materialsStart) / 48] @ materialsStart + 48;
float additionalChoices[4] @ choicesStart + 48;
Guid eyeColor;
Guid secondEyeColor;
Guid hairColor;
};
So the next major issue would be the nature of the hash/checksum, a topic I am anything but familiar with.
Comparing the component index across saves from different patches has reinforced my guess that the last unknown 8 byte section is related to the component name, since it always changes when a component gets renamed, for example because it got a new version.
Given that they have the same size as the file checksum and were probably developed alongside the main header, I think it's plausible they'd use the same algorithm, which would mean much more manageable data pairs to work with.
For some reason, newage also partially contains some WPF code? This was taken from one of my recent saves (patch 3).
Not sure if WPF specifically, but this is definitely XAML, including indentation (all the 0x20). Here's the sample rendered:
lication:,,,/GustavNoesisGUI;component/Assets/CharacterSheet/btn_round_medium_h.png"/>
</Trigger>
<Trigger Property="IsPressed" Value="True">
<Setter TargetName="bg" Property="Source" Value="pack://application:,,,/GustavNoesisGUI;component/Assets/CharacterSheet/btn_round_medium_p.png"/>
</Trigger>
<Trigger Property="IsChecked" Value="True">
<Setter TargetName="bg" Property="Source" Value="pack://application:,,,/GustavNoesisGUI;component/Assets/CharacterSheet/btn_round_medium_p.png"/>
</Trigger>
</ControlTemplate.Triggers>
</ControlTemplate>
</ls:LSToggleButton.Template>
</ls:LSToggleButton>
<Popup IsOpen="{Binding IsChecked, ElementName=ToggleBtn}" Placement="Bottom" StaysOpen="False" HorizontalOffset="-40">
<ls:LSNineSliceImage x:Name="PopularFiltersHolder" Margin="0,-12,0,0" HorizontalAlignment="Stretch" ImageSource="pack://application:,,,/GustavNoesisGUI;component/Assets/CharacterPanel/sorting_bg.png" Slices="60" Padding="10,40">
<StackPanel Margin="30,0,40,0" IsItemsHost="True" MinWidth="200" MinHeight="200"/>
</ls:LSNineSliceImage>
</Popup>
</Grid>
</ControlTemplate>
</ComboBox.Template>
<ComboBox.ItemContainerStyle>
<Style TargetType="ComboBoxItem">
<Setter Property="Template">
<Setter.Value>
<ControlTemplate TargetType="ComboBoxItem">
<Grid x:Name="ButtonRoot" Background="Transparent">
<Image x:Name="HLBG" Source="pack://application:,,,/GustavNoesisGUI;component/Assets/CharacterPanel/selector_listitem_d.png" Str
For those who have not worked with WPF, this is a sample of code for defining the style and behaviour of a ComboBox ( i.e drop-down menu) and the items it lists.
Looking at the namespace references (e.g: pack://application:,,,/GustavNoesisGUI;component/Assets/CharacterPanel/sorting_bg.png ), this XAML is from a project that uses another project GustavNoesisGUI in the same Visual Studio solution.
Judging by the level of indentation, it's part of a big view or a complex component.
Now, why this source code sample is in NewAge, including indentation, and incomplete... ??? If that had been a personal project and I saw that, I'd say I'd have made a mistake crawling directories to fetch a specific byte range in all files. But in a save file?
Interesting, it is part of the game file Public\Game\Gui\Widgets\PartyPanel.xaml, not sure how it got in the save though.