Fix ludios_wpull to support SQLAlchemy 1.4

[ivan]

trafficstars

Please help: I want a working grab-site in nixpkgs master, but nixpkgs has SQLAlchemy 1.4 and wpull does not support it (https://github.com/ArchiveTeam/wpull/issues/463). I would like the same issue fixed in ludios_wpull, but I haven't looked into it.

[TheTechRobo]

On a sort-of offtopic note, what is the status on merging ludios_wpull into wpull?

[ivan]

I would certainly be happy if someone took on that project, as I have no time to maintain grab-site at the moment.

I forked to ludios_wpull when it looked like wpull would not be maintained, and I changed and removed stuff without coordination with JAA or ArchiveTeam. I haven't looked at it in while and don't know what the results of a merger should look like.

[matthewcen]

I added basic SQLalchemy 1.4 support (amongst other upgrades) in my fork here: https://github.com/matthewcen/ludios_wpull/commits/master

The specific commit that added the comparability is here: https://github.com/matthewcen/ludios_wpull/commit/5c3b167c00ee37cfabe4866d32da7fa9dd6f2946

Please note i have only done very basic testing (e.g. changed enough code to get SQLAlchemy 1.4 to stop throwing errors for the various crawls I was performing)

[ivan]

@matthewcen Thanks for your work there. Do you think https://github.com/matthewcen/ludios_wpull ready to merge into ludios_wpull? I am about to begin testing it because Python 3.8 is running out of time.

[ivan]

https://github.com/matthewcen/ludios_wpull/commit/10854d200dc4d4171b802664f976ba8f7292d2b5 removed the wpull version from the WARCs, but we need the version to be able to identify possibly malformed WARCs in the future if they are caused by specific wpull versions.

[matthewcen]

@matthewcen Thanks for your work there. Do you think https://github.com/matthewcen/ludios_wpull ready to merge into ludios_wpull? I am about to begin testing it because Python 3.8 is running out of time.

Hi, @ivan , I will probably need to do some cleanup/touchup but it should be ready for testing.

matthewcen/ludios_wpull@10854d2 removed the wpull version from the WARCs, but we need the version to be able to identify possibly malformed WARCs in the future if they are caused by specific wpull versions.

I will need to review to determine why it was removed. But it can be readded easily.

[HeliosLHC]

Fixed upstream in ludios_wpull version 5 which supports SQLAlchemy 2.0+