What to do about "left-padded" packages (packages for which the Git repo has been deleted)

[StefanKarpinski]

https://github.com/JuliaComputing/JuliaHub/issues/67

[DilumAluthge]

We could remove those packages from the General registry.

They'll still be installable from the Pkg server.

[DilumAluthge]

DecentralizedInternet is a false positive. The Julia package is in a subdirectory, which is allowed.

https://github.com/Lonero-Team/Decentralized-Internet/tree/master/.julia-configs

https://github.com/JuliaRegistries/General/blob/6626603a1d9df7982b558e9b4b16d58d2f8ac7da/D/DecentralizedInternet/Package.toml#L1-L4

[fredrikekre]

We could remove those packages from the General registry. They'll still be installable from the Pkg server.

But the PkgServer serves the registry so it has to be in there.

[DilumAluthge]

Well, if you have a Manifest.toml file, you can instantiate it without the registry, right?

[fredrikekre]

Yes.

[DilumAluthge]

In that list of packages that have been deleted on GitHub, we should subdivide the list into:

1. Packages that have no dependents in the General registry
2. Packages that have one or more dependents in the General registry

Then we can decide what to do with each of those two categories separately.

[Gregliest]

Alright I did a first pass on analyzing the dependencies. First, the list that I generated and the list from the original issue are different. I did a spot check on the differences, and the extra packages that I checked from the original list are not currently in this registry. Examples include: https://github.com/JuliaComputing/Blpapi.jl.git https://github.com/JuliaComputing/MiletusPro.jl.git https://github.com/aramirezreyes/SAMtools.jl.git Also note that https://github.com/Lonero-Team/Decentralized-Internet.git appears to be working properly.

So, I'm assuming that the list I generated today is correct: https://github.com/MUsmanZahid/TeXTable.jl https://github.com/PetrKryslUCSD/MeshPorter.jl https://github.com/mipals/SymSemiseparableMatrices.jl https://github.com/owinter92/SimpleTweaks.jl https://github.com/rjdverbeek-tud/Atmosphere.jl https://github.com/anders-dc/Granular.jl https://github.com/StanJulia/StanMCMCChain.jl https://github.com/slmcbane/MirroredArrayViews.jl https://github.com/mrtkp9993/Bioinformatics.jl https://github.com/JonathanCourtois/MackeyGlass.jl https://github.com/GuilhermeHaetinger/KelvinletsImage.jl https://github.com/bcbi/CountdownNumbers.jl https://github.com/MasFlam/Tui4jl https://github.com/PetrKryslUCSD/MeshKeeper.jl https://github.com/markushhh/YahooFinance.jl https://github.com/louiscmm/Sinaica.jl https://github.com/PetrKryslUCSD/MeshMaker.jl https://github.com/bcbi/CountdownLetters.jl https://github.com/tlienart/MLJScikitLearn.jl https://github.com/Geographs/AuthGG.jl https://github.com/MUsmanZahid/DTALib.jl https://github.com/extradosages/RestApis.jl https://github.com/PetrKryslUCSD/MeshFinder.jl https://github.com/Moelf/BigG.jl https://github.com/atoptima/HighLevelTypes.jl https://github.com/rbalexan/InfrastructureSensing.jl https://github.com/lucianolorenti/Estapir.jl https://github.com/oscar-system/GAPTypes.jl

Of these, I only found two with dependencies: StanMCMCChain -> 2 dependencies (StatisticalRethinking, StanModels) GAPTypes -> 1 dependency (GAP)

The rest didn't have any dependencies that I found.

Here's the script that I used. It's quick and diry, and feedback is welcome. I also hope it works haha, I didn't want to rerun it from the start after cleaning it up, due to the Github api interaction and rate limiting.

[DilumAluthge]

The two deleted packages in the https://github.com/bcbi organization can safely be deleted from the General registry. In hindsight, those two packages should not have even been registered in the first place. That's my fault.

[StefanKarpinski]

Want to go ahead and make a PR to delete those?

[StefanKarpinski]

We should probably go ahead and delete all left-padded packages and see if anyone complains. If they don't, then problem solved.

[fredrikekre]

Why is it a problem that they are in here?

[StefanKarpinski]

There's no way to install them, so what's the point?

[fredrikekre]

You can through the PkgServer though. Wasn't that one of the motivations for the PkgServer infrastructure, that you could still install left-padded packages?

[StefanKarpinski]

Any that can be installed through the PkgServer we should keep but I think some of these are not cached. Those are the ones I think we should get rid of.

[fkastner]

Is there an official list or somewhere we should report deleted packages? Also, even though one can still add them, deving them fails because that looks for the git repository. Would it make sense to record in the registry if a repository was deleted? Maybe remove the repo link in the Package.toml?

[tpapp]

It is not clear to me if PkgServer is a backup or a cache.

If it is a backup, in the sense that all version of all packages ever registered (in the general registry) remain available, then keeping them is fine.

If it is a cache, users of disappearing packages should learn ASAP, so removing after a grace period (eg 3 days, notify the author so that unintended technical glitches can be fixed) seems reasonable. It would then trigger CI failures etc.

[StefanKarpinski]

See https://github.com/JuliaLang/Pkg.jl/issues/1377. The pkg servers are caches; the storage servers are persistent backup. So the system as a whole can server anything it has served at some point forever.

[ViralBShah]

Closing as I'm not sure if there's anything further to do here, but please reopen if necessary.