Linking related files with a `.from_archive()` `__init__`

[snake-biscuits]

From #191: We should also look at some standard method for linking related files

sega.Gdi, respawn.RPak, respawn.Vpk & valve.Vpk could all use this too since their .read() methods can't function without grabbing data from other files

@classmethod
def from_archive(cls, archive: base.Archive, filename: str) -> Union[ArchiveClass, Bsp]:
    out = cls.from_bytes(archive.read(filename))
    for related_filename in cls.related_filenames(archive.namelist()):
        out.link(archive, related_filename)
    return out

Places where this'd be useful

ArchiveClasses

• [ ] respawn.RPak
• [ ] respawn.Vpk
• [ ] sega.Gdi
• [ ] valve.Vpk

BspClasses

• [ ] ValveBsp
• [ ] RespawnBsp (respawn.ExternalLumpManager)

NOTE: all BspClasses could use a .related_filenames() class

Related

• #156

[snake-biscuits]

really, this has 2 parts The listed BspClasses have external files we parse But .link can be in base.Bsp

.from_archive will likely be per class

[snake-biscuits]

started looking at this now archives.base.Archive has a .from_archive @classmethod __init__

planning to add .mount & .unmount methods to base.Archive to link "external files" these can provide some kind of file handles to keep the data present

.unmount will be to reduce memory usage unmounting a file that was added automatically will probably be really painful

[snake-biscuits]

implementing .mount for BspClasses will likely be more complex since RespawnBsp has ExternalLumpManager to try and cache files

we might need to keep the archive open to dynamically mount .bsp_lump

bsp.mount(archive, filename) makes sense but I'd also like to mount from files, streams & bytes

might be a while until I land on an implementation

[snake-biscuits]

.link can be in base.Bsp

.from_archive will likely be per class

.from_archive has been implemented in base.Bsp If we have a .related_filenames() function we can use that to automatically check for & mount files This should probably be optional & off by default to reduce memory usage

Some RespawnBsp (Apex Season 10 onwards) will need external .bsp_lump to be useful though

We already have .mount_lump methods, so I'm thinking .mount_file should work as a name These should do the job:

def mount_file(self, filename: str, archive=None):
    if archive is None:
        self.external_files[filename] = open(filename, "rb")
    else:
        self.external_files[filename] = io.BytesIO(archive.read(filename))

def unmount_file(self, filename: str):
    self.external_files.pop(filename)

Afaik all external files would be binary If not we can always add plaintext=False to .mount_file

Could be useful to have some kind of .stream method in ArchiveClasses to reduce memory usage Likely a lot slower and would have to be tailored to each individual ArchiveClass Maybe keep it in mind for if we come back to optimise this system

[snake-biscuits]

particle manifests would be a plaintext extra file for ValveBsp though as discussed in #156, we're going to focus on mounting lump data (extra lighting information & external lumps)

[snake-biscuits]

.related_filenames isn't going to work

.lmp filenames include version numbers[^vdc] we should probably use fnmatch patterns instead this could be used with archive.namelist() or os.listdir() though we'll need some path splitting solution for working out mod-relative paths

path_tuple in archives.base could be useful probably handy for autodetect.naps too maybe we add a new filesystem module?

[^vdc]: Valve Developer Community: Pathing levels with lump files

[snake-biscuits]

we could extract files we want to mount inside archives if we use temporary files we'd be reducing ram usage load time would only increase by the write time we have to get every byte of the file from the archive either way

until we work out a system for streaming assets in archives anyway implementations of streaming would be unique to each ArchiveClass io.BytesIO(archive.read(filename)) might work as a default

[snake-biscuits]

extras seems like a good name to reference "external files" helps keep variable names short (mount_extras vs. mount_external_files)

totally didn't get the idea from the Guilty Gear soundtrack playing in another tab

[snake-biscuits]

ArchiveClasses & BspClasses now automatically mount extras

each specific class which uses external files now needs to access them via self.extras they also need class-specific .extra_patterns implementations

[snake-biscuits]

We need archives we can use to actually test file mounting I currently don't have any .bsps inside archives w/ supplementary files

Might have to make an id_software.Pak w/ .lit & .vis files Can't use respawn.Vpk because we don't have compression yet

Tho I guess for ArchiveClasses I could test a sega.Gdi inside a pkware.Zip

[snake-biscuits]

Could add a raw bytedata arg to .mount_file for testing

def mount_file(self, filename: str, archive=None, raw_data: bytes = None):
    if raw_data is not None:
        self.extras[filename] = io.BytesIO(raw_data)
    else:
        ...  # load from file or archive as before

I'm sure this could also be useful outside of testing

[snake-biscuits]

ArchiveClasses need to start using filename & folder attributes This will lets us generate extra_patterns & scan the local folder for attached files

[snake-biscuits]

Not every ArchiveClass needs to track folder & filename .from_archive & .from_file handle that for us anyway

[snake-biscuits]

got sega.Gdi implemented but to test it, I tried updating sega.GDRom to read the data unfortunately the .gdis I'm using don't use a single track for the GD-ROM Area

base.Archive is not at all the right tool to map disc-related formats really need to make a new type of object for presenting a virtual disc and initialising cdrom.Iso from that (while maintaining .iso file compatibility)

NOTE: cdrom.Iso is an ArchiveClass, since it has an actual filesystem

would also reduce memory costs by "streaming" data from discs

[snake-biscuits]

DiscClasses use external.File to hold onto track data