Revisit shadow repo handling

[AMDmi3]

Currently some repos are marked as shadow, which means that if a metapackage exists only in shadow repos, it's hidden from the list. The purpose is to avoid noise from e.g. cpan or rubygems repos, which have tens of thousands modules not packaged elsewhere. It's also used to hide windows-only software coming from chocolatey and scoop repos and a lot of splits from openwrt.

The way we handle it is inconsistent though. Shadow packages are:

• hidden from the list (ok for general list, probably not good for list of repository/maintainer metapackages)
• not accounted in the statistics (not good; shadow should affect metapackages, not packages which should be counted)
• shown on maintainer page and counted in maintainer statistics (good, apart from the inconsistency with previous items)

Also, it's now possible to track when a metapackage appears or disappears from repology sight, and this requires even more complex logic wrt shadow: if we track all packages, feed will mostly show activity in the package repos. If we don't show them, how do we treat metapackages removed from normal repos but still present in shadow ones?

This should be thought over, and I'd prefer to simplify logic by either ignoring all shadow-only packages, or treat them as normal ones. The former approsch creates noise though, while the latter crestes more inconsistency.

[AMDmi3]

Update: more windows and macosx specific repositories have emerged, which suggests the following idea: turn shadow flag into namespace-like thing.

• Each repository may have a namespace defined
• Default namespace is assumed to be "cross-platform software"
• If all packages for a project have the namespace defined to the same value, whole project falls into this namespace
• The website allows to exclude specific namespaces from the listing

[noraj]

I have projects packaged in some linux distro but that are shadowed and so not displayed in rubygems or pypi. It seems The projects appeared on rubygems or pypi first, were shadowed, then appeared in a linux repo but were never unshadowed afterward even if they are packaged elsewhere now.

Examples:

• https://repology.org/project/compp/versions and https://pypi.org/project/ComPP/
• https://repology.org/project/pass-station/versions and https://rubygems.org/gems/pass-station (https://repology.org/project/ruby:pass-station/versions)

[AMDmi3]

The projects appeared on rubygems or pypi first, were shadowed, then appeared in a linux repo but were never unshadowed afterward even if they are packaged elsewhere now.

It does not work this way. compp and python:compp are different projects, and python:compp is hidden because it only comes from pypi. The solution is to merge them.