Add functions for working with entity lumps

[nosoop]

With the merge of #1534 there is currently no way to access the level entity string. LevelKeyValues among others was built around having that ability.

Here's my proposal for a new API to deal with entity lumps. @asherkin thoughts?

(I'll be writing a draft implementation of it as a SourceMod extension, so input on the API design is greatly appreciated.)

/**
 * A handle to a weakref of a list of key / value pairs; any write operations on this handle will affect the resulting map entity string.
 * If the entry in the EntityLump is removed, the handle will error on all operations.  (The handle will still need to be deleted.)
 */
methodmap EntityLumpEntry < Handle {
	/**
	 * Copies the key / value at the given index into buffers.
	 *
	 * @error Index is out of bounds.
	 */
	native void Get(int index, char[] keybuf = "", int keylen = 0, char[] valbuf = "", int vallen = 0);
	
	/**
	 * Updates the key / value pair at the given index.
	 * NULL_STRING may be used to preserve the existing value.
	 *
	 * @error Index is out of bounds or entity lump is read-only.
	 */
	native void Update(int index, const char[] key = NULL_STRING, const char[] value = NULL_STRING);
	
	/**
	 * Inserts a new key / value pair at the given index, shifting the existing entry and ones past it up.
	 * If EntityLumpEntry.Length is passed in, this is an append operation.
	 *
	 * @error Index is out of bounds or entity lump is read-only.
	 */
	native void Insert(int index, const char[] key, const char[] value);
	
	/**
	 * Removes the key / value pair at the given index, shifting all entries past it down.
	 *
	 * @error Index is out of bounds or entity lump is read-only.
	 */
	native void Erase(int index);
	
	/**
	 * Inserts a new key / value pair at the end.
	 *
	 * @error Index is out of bounds or entity lump is read-only.
	 */
	native void Append(const char[] key, const char[] value);
	
	/**
	 * Returns the next index with the matching key, or -1 if not found.
	 *
	 * NOTE: Just a convenience function; probably fine to omit in favor of .Get()
	 */
	native int FindKey(const char[] key, int start = -1);
	
	/**
	 * Returns the number of key / value pairs in the list.
	 */
	property Length {
		native int get();
	};
};

/**
 * A group of natives for a singleton entity lump.
 * EntityLumpEntry instances are only available for writing during OnMapInit().
 */
methodmap EntityLump {
	/**
	 * Returns the EntityLumpEntry at the given index.
	 * This handle should be freed by the calling plugin.]
	 *
	 * @error Index is out of bounds.
	 */
	static native EntityLumpEntry Get(int index);
	
	/**
	 * Erases an EntityLumpEntry at the given index.
	 * Any handles referencing that EntityLumpEntry are invalidated.
	 *
	 * @error Index is out of bounds or entity lump is read-only.
	 */
	static native void Erase(int index);
	
	/**
	 * Inserts an empty EntityLumpEntry at the given index.
	 *
	 * @error Index is out of bounds or entity lump is read-only.
	 */
	static native void Insert(int index);
	
	/**
	 * Creates an empty EntityLumpEntry, returning its index.
	 *
	 * @error Entity lump is read-only.
	 */
	static native int Append();
	
	/**
	 * Returns the number of entities currently in the lump.
	 */
	static native int Length();
};

[nosoop]

Implementation of the above API available here.

As of this comment edit, it's capable of dealing with:

• lumps using exotic line endings (a L4D2 map that was made official only uses \r)
• lumps containing values spanning multiple lines
• bhop_miku_v2, containing a ~15MB entity lump (on my machine with standalone parser utility + compile-time optimizations enabled, it takes ~~about 2.5s~~ less than 0.5s to parse)
• all of the entity lumps I've batch extracted via hlextract from the community TF2 maps I've downloaded over the years (749 in total, aside from the dozen or so that either hlextract or the LZMA decompression routine failed on)
  • round trip testing using diff -Baw decompressed/ parsed/ indicated no differences in non-whitespace sequences aside from one entity lump string with many null bytes at the end
• gracefully failing to some extent if the lump couldn't be parsed successfully (it will skip calling the forward and will not replace the string)

If there isn't a suitable replacement for accessing / modifying the entity lump once 1.11 reaches stable (whether it's this solution or something else), I'll go ahead and maintain the extension myself.

... that may actually be better, since it would also work on oldstable without conflicts.

[Ilusion9]

Isn't too late to change entity lump after the entity is created? (your sourcepawn example plugin) I think the changes should be done only in OnLevelInit, not after that.

@asherkin can retrieving the entities lump as a KeyValue or StringMap + ArrayList (example here https://github.com/Ilusion9/mapentites-sm/blob/main/scripting/mapentities.sp) crash the server if the size is greater than 2MB?

[nosoop]

Yes, the extension currently allows for writing to entity lump handles (the read-only exception logic is currently unimplemented, but it would be straightforward to implement — I'm just lazy). The string conversion only happens immediately after the forward is called, so modifications won't actually have any effect outside of that forward.

OnMapEntitiesParsed is provided since the extension can't listen in on SourceMod's OnLevelInit / OnMapInit to determine if the hook needs to be called with modified parameters. (This won't be necessary if the implementation gets merged into core.)

The plugin you've listed no longer works with SM 1.11 as #1534 removes support for reading / modifying the entity lump string entirely (also see #1470). The forward receives an empty string instead; plugins have no way to parse it (short of manually calling IVEngineServer::GetMapEntitiesString()).