EventStore Bloat/Leaking?

[ect0s]

(https://github.com/Team-Silver-Sphere/SquadJS/blob/cdf3a1e36d9c051c3041827019a17836f56bde24/squad-server/log-parser/deployable-damaged.js#L16)

Is this ever cleared from the event store, and if so, where?

Doing some testing on a persistent log parser, and dumped the event store after parsing a log only to find alot of data sitting there and never getting cleared.

[Thomas-Smyth]

I don't believe that's currently cleared anywhere. We should probably wipe some event store data on layer changes, but we need to be careful that we don't wipe data that is required across layer changes. That might mean we have to split the event store into sub objects for each different type of data being stored.

[ect0s]

I've started doing this for the changes I'm making, its going to be a bit of work.

Theres a few other places where I'm not sure what the eventStore is trying to do, in particular the unrolls in

wounded, died, damage, and revived.

I was testing, and PLAYER_REVIVED never has an attackerName, or very rarely does.

https://github.com/Team-Silver-Sphere/SquadJS/blob/cdf3a1e36d9c051c3041827019a17836f56bde24/squad-server/log-parser/player-revived.js

https://github.com/Team-Silver-Sphere/SquadJS/blob/cdf3a1e36d9c051c3041827019a17836f56bde24/squad-server/index.js#L261

I've managed to disambiguate PlayerControllers in Died and Wounded via the eventStore; instead of sending these up the chain I replace them with a lookup for the PlayerSuffix. This is done via two new events related to player controllers that set a

logParser.eventStore.playerControllers[BP_PlayerController_C_...] = suffix

Clearing this will require some iteration on PlayerDisconnected, however, we likely want a ~3 minute timer (or whatever max bleedout time is) so that events coming after can coalesce. For example, if I wound a player, then disconnect, then he bleed out. I need to 100% test that this path happens, because this might be a source of 'nullptr' events

This is where I noticed the issue with bloat, because I knew I wasn't clearing my values and I wanted to see what happened with my ~400k line test log.

[Thomas-Smyth]

The unpacking in the wounded, died and revived lines is done to carry forwards information only accessibly in the previous event, I think in particular the attacker's information which is only accessible via the damaged line if I recall correctly and the time of wounding to died/revived.

I've managed to disambiguate PlayerControllers in Died and Wounded via the eventStore; instead of sending these up the chain I replace them with a lookup for the PlayerSuffix.

I don't quite get what you're talking about, but we should avoid using the player suffix at all costs as it's probably the most inaccurate part of SquadJS. I think now player roles and admin camera usage are accessible via RCON there's probably an argument for removing the PLAYER_POCESSES event and removing the player suffix logic that would probably no longer be required after that.

I would avoid complex stuff for removing things too aggressively as otherwise we'll have a lot of headaches just for the purpose of saving a tiny bit of memory.

[ect0s]

So, heres the deal:

DBlog has alot of null data because we fail to lookup via PlayerControllers in events where they replace a player name.

In the current version of what I have worked on, if a unique match is found from Suffix, its good data that we currently don't have.

If we fail, well, the path is the same as it currently is, where we don't fill out data in those events and end up with null in the DB.

This is done via checking for >1 match in getPlayerByCondition, in which case return null.

[ect0s]

Before {
  raw: '[2022.04.08-22.31.59:904][826]LogSquadTrace: [DedicatedServer]Wound(): Player:popper69 KillingDamage=40.000000 from BP_PlayerController_C_2147458718 caused by BP_L85A2AG36_ElcanLDS_Rifle_C',
  time: 2022-04-08T22:31:59.904Z,
  chainID: 826,
  victimName: 'popper69',
  damage: 40,
  attackerName: 'hawk32791',
  weapon: 'BP_L85A2AG36_ElcanLDS_Rifle',
  attackerPlayerController: 'BP_PlayerController_C_2147458718'
}
After {
  raw: '[2022.04.08-22.31.59:904][826]LogSquadTrace: [DedicatedServer]Wound(): popper69 KillingDamage=40.000000 from BP_PlayerController_C_2147458718 caused by BP_L85A2AG36_ElcanLDS_Rifle_C',
  time: 2022-04-08T22:31:59.904Z,
  chainID: 826,
  victimName: 'popper69',
  damage: 40,
  attackerName: 'hawk32791',
  weapon: 'BP_L85A2AG36_ElcanLDS_Rifle',
  attackerPlayerController: 'hawk32791'
}

[ect0s]

If we don't want to do it via suffix, I think I can replace those with steamIDs. I'm already handling player connected and steamid being out of order and merging the information but that code is a sin right now.

Something like a table where SteamID => Suffix inside the eventStore. But then, we also need a way to remove this.

Then we would need to check for a name/steamid in the getPlayerByCalls.

The real issue inside of the eventStore is needing lookups via 3 ways; SteamID, Suffix, PlayerController.

{
  raw: '[2022.04.08-22.46.22:973][227]LogSquadTrace: [DedicatedServer]Wound(): Player:Tybo KillingDamage=192.000000 from BP_PlayerController_C_2147378572 caused by BP_L85A2_ElcanLDS_Foregrip_C',
  time: 2022-04-08T22:46:22.973Z,
  chainID: 227,
  victimName: 'Tybo',
  damage: 192,
  attackerName: 'Hurkysword',
  weapon: 'BP_L85A2_ElcanLDS_Foregrip',
  attackerPlayerController: '76561198...'
}

attackerPlayerController is now a steamID

[ect0s]

I also notice with Damage, Wound and Die we never reap the eventStore[playerName]

I think RestartPlayer() is a respawn, which might make it possible; unfortunately its the playerSuffix that appears in this line.

Dumping a full log and parsing the eventStore has given me a ~160kbyte file for the event store; so this might not actually be a big deal; however the eventstore is unbound by runtime so this leak will get progressively worse as uniquely named players join and leave the game, get damaged, and set keys that are never cleared.

[ect0s]

Alternatively, inside playerDisconnected, we have a steamID, a Full Name, and a Suffix.

These should let us reap the eventStore eventually.

This works for a running SquadJS instance with RCON state, but from a "logs only" perspective its difficult

[Thomas-Smyth]

It would just be easier to wipe those at the start of a new game as damage, wounds. deaths don't carry across between games. Like I said before, there's not much point in coming up with a massively complex system to clear out the store if it's only going to save a tiny amount of memory.

[ect0s]

That works.

I was trying to sort the player controller situation out in the log parser, and needed that to persist across maps.

Player Controllers are unique to each player and each instance of them joining the server.

    this.eventStore = {
      playerControllers: {'Controller': 'Name'}
      pendingPlayers:{}
      disconnected:{steamID: Player{...}}
    };

  getPlayerFromController(controller){
    const matches = this.eventStore.playerControllers[controller]
  
    if (!matches){
      //this should never happen. 
      return controller
    }
    return matches[0];
  }

As for reaping them; it is possible to capture playercontrollers before emiting a player connected event, even in an error path where steamid is delayed. I've been testing this and throwing uncompleted players into the 'pendingPlayers' property. These are objects where

{
suffix: Suffix
steamID: steamID
controller: controller
}

I've been testing on multiple logs, and so far this is infallible. These get removed when a player finishes joining.

onDisconnect, I've been setting these disconnected steamIDs into eventStore.disconnected, so they can be reaped, since this state is going to continue over match to match.

This can be done on a timer, or in my case, since I'm parsing offline logs, I've waiting till the logparser has fully completed. The idea here is for persistently parsing the logFile on startup each time. Even on large files, the delay is seconds with taillogs. With FTP the download speed will govern how long this "startup' phase lasts, which is already the case with parsing log events over FTP.

For the rest of the data, this can be flushed on each map change,

So we likely would want to set these in something like

eventStore.mapData.[whatever is already called in events]

And on NEW GAME, we just delete/reset eventStore.mapData to an empty object.

[ect0s]

We fire 'PLAYER_DISCONNECTED' when EAC unregisters a client, which happens on map changes.

But this event doesn't actually mean the player disconnected; as they may reregister for the new map.

STEAMID_CONNECTED

grep 'RegisterClient' SquadGame-backup-2022.04.09-07.36.18.log | wc -l
2017

PLAYER_DISCONNECTED

grep 'Unregister' SquadGame-backup-2022.04.09-07.36.18.log | wc -l
2013

PLAYER_CONNECTED:

grep 'Join succeeded' SquadGame-backup-2022.04.09-07.36.18.log | wc -l 
637

[ect0s]

https://github.com/ect0s/SquadJS/blob/Player-Disconnected-Fix/squad-server/log-parser/player-disconnected.js

Found a better regex for player disconnected