project files filtering

[pmp-p]

as @davidpendergast suggested on discord we should have some kind of .pygbagignore file ( following .gitignore convention but not limited to )

to integrate in the pipeline

actually pack.py does too much things it should be splitted

[ gathering.py ] <- read a path to a zip or a folder.

unpack a project archive or take a source folder and search for .gitignore and .pygbagignore and merge them each time.

-> save a .pygbagignore at the unpacked project root -> produce an iterator of valid files to use

[ filtering.py ] <- read a list

apply the merged .pygbagignore saved file.

apply heuristic to detect harmfull assets like wav / oversized png / mp3 unreadable files offer choices report about potential download time of game on first use.

update the .pygbagignore with comments line starting with # to specify optim applied to file or wildcard set of files (re-using the same parser logic of gitignore blocklisting for a different purpose).

-> produce a iterator of tuple of valid files to use in output each annotated with optimization to apply ( maybe not yet available but still ... )

[ optimizing.py ] <- read tuple list

apply optim where possible ( https://github.com/scour-project/scour , pngquant, ffmpeg convertions etc ) remove docstrings. use https://github.com/interpreters/pypreprocessor to remove platforms dead code

-> produce a iterator of valid files to pack.py

future: when running in browser pygbag should offer to download the .pygbagignore file if created/changed

[davidpendergast]

This sounds great, couple questions though:

unpack a project archive or take a source folder and search for .gitignore and .pygbagignore and merge them each time.

Is the .gitignore relevant here? I'd say a .pygbagignore alone is enough - if we automatically ignore the contents of your .gitignore, you could end up in a situation where you can't archive something (that you want to archive) because it's gitignored (e.g. some kind of generated file) and you'd have to do something awkward like temporarily un-gitignore it when you're doing the web packaging to work around that. Seems like the potential convenience (of not having to update your .pygbagignore) isn't worth it.

save a .pygbagignore at the unpacked project root

Should the tool be writing to this file at all (aside from the first run, maybe)? I'd argue no - this should be something the user has complete control over and can update/not update as they desire?

apply heuristic to detect harmfull assets like wav / oversized png / mp3 unreadable files offer choices

Maybe it could just show a warning? If you're purposely trying to include something the script disapproves of, with this approach it'd complain about it every time you run a build which seems unnecessary?

report about potential download time of game on first use.

Is this possible to do? Wouldn't this depend on a number of factors (connection speed, computer speed)?

update the .pygbagignore with comments line starting with # to specify optim applied to file or wildcard set of files.

Hmm, you're saying the script will reach into the user's file and add comments to things? Or add new lines of not-actually-ignored things (because they're really just getting converted into something else)? I'm a bit confused by this, seems like it deserves its own config file separate from the ignorelist?

[pmp-p]

Is the .gitignore relevant here

you are 100% right, but i have reasons for doing that, so to force un-ignoring a .gitignore directive .pygbagignore should have a directive :)

Should the tool be writing to this file at all

yes and everytime or next build/packing could be different when changing cdn or python version. if user edit it manually, changes will be merged next run anyway and stay.

Maybe it could just show a warning

filtering should be strong, and able to block publishing ( mp3 for example is NOT readable on some major browsers). also some files like .so must me tested if valid wasm and not linux stuff that landed there by mistake. it's also possible in the future that we are required by law to check usage license on assets via some online service. but don't worry apart from those extremes cases "offer choices" just means for me "read the warning, the suggestions and do what you want/need to do, then relaunch"

Is this possible to do ?

the minimal download time is possible to calculate for typical because pygbag knows the size of the cache right after the first run, so giving calculation for adsl connections/ 3G/4G, regardless of added time with decompression it's a good reminder for people connected via optical network/5G that not everyone is so happy in the world or on mobile data plan.

Hmm, you're saying the script will reach into the user's file and add comments to things?

i hope in the future people can put # "optimize next lines that way" comments themselve in the .gitignore to be merged on first run because pygbag has get so popular XD . So pygbag showing how to write those hints seems a good idea ( though it may not possible to actually run those optim depending on host platform, hence the "comment" notion )

( still not clear in my mind though exactly how those "hints" would fit with .gitignore ) maybe eg

# pngquant --quality 40
# /data/reallybig/img/

# wav2ogg
# /data/reallybigwav/

/keystore
*.map
*.elf
*.bin
*.rej
*.orig
*.o
*.a
*.egg-info
*.pyc

[davidpendergast]

I see, so it sounds like this file is going to be handling a lot of things (files to ignore, optimization configs, a flag about whether to respect the .gitignore (perhaps)), making it more of a general .spec file than a dedicated ignorelist file? I assume more things will be added in the future as well.

So that makes total sense to me. Here would be my suggestion though (about the file merging stuff):

• Let the user have their own spec file, mypygbag.spec, which the script never modifies. Here they can configure their permanent ignorelist and override other default settings. This file could be checked into version control and shared among developers.
• Whenever pygbag is launched, it generates a fresh pygbag.spec file (which merges things from the user's spec with its own default settings). This is what the script actually uses while packaging, and this would not be useful to check into version control.

This way, you could run the pygbag command as much as you want without permanently modifying your specs every time.

So then it's just a matter of designing the format of the file, the merging procedure, and the settings that belong in it (which it sounds like you have ideas about already).

[pmp-p]

we also need a font cache builder as a helper for pygame runtime : since it cannot run fc-list on browsers https://github.com/pygame/pygame/pull/3225#pullrequestreview-1087894000 https://github.com/pygame/pygame/issues/3432