Content import composability refactor and single resource import

[rtibbles]

Summary

• Adds the ability to import content metadata from JSON rather than a SQLite database
• Updates SQLAlchemy version to 1.4 and makes syntax updates
• Allows for import of partial channel data (sub trees rather than the whole tree) and incremental import of the entire tree
• Creates a public API endpoint to return the required metadata JSON for importing a single resource
• Does some refactors to make content import functionality more composable
• Creates an async task for importing a single resource over the network

References

Fixes #9552 Fixes #9492 Fixes #9491 Completes some of #9266 Last commit also fixes https://github.com/learningequality/kolibri-app/issues/105, which might be cherry-picked to https://github.com/learningequality/kolibri/pull/9683 if that is approved for merge first.

Reviewer guidance

Do the tests of the new import functionality make sense? Does importing a resource from another instance work?

Does the way I've split up the different functionality of the import manager classes feel readable/practical/sustainable?

---

Testing checklist

• [ ] Contributor has fully tested the PR manually
• [ ] If there are any front-end changes, before/after screenshots are included
• [ ] Critical user journeys are covered by Gherkin stories
• [x] Critical and brittle code paths are covered by unit tests

PR process

• [x] PR has the correct target branch and milestone
• [x] PR has 'needs review' or 'work-in-progress' label
• [x] If PR is ready for review, a reviewer has been added. (Don't use 'Assignees')
• [ ] If this is an important user-facing change, PR or related issue has a 'changelog' label
• [ ] If this includes an internal dependency change, a link to the diff is provided

Reviewer checklist

• Automated test coverage is satisfactory
• PR is fully functional
• PR has been tested for accessibility regressions
• External dependency files were updated if necessary (yarn and pip)
• Documentation is updated
• Contributor is in AUTHORS.md

[github-actions[bot]]

Build Artifacts

Asset type	Download link
PEX file	kolibri-0.16.0.dev0_git.20220927205947.pex
Unsigned Windows installer	kolibri-0.16.0.dev0+git.20220927205947-unsigned.exe
Debian Package	kolibri_0.16.0.dev0+git.20220927205947-0ubuntu1_all.deb
Mac Installer (DMG)	kolibri-0.16.0.dev0+git.20220927205947-0.3.0.dmg
Source Tarball	kolibri-0.16.0.dev0+git.20220927205947.tar.gz
WHL file	kolibri-0.16.0.dev0+git.20220927205947-py2.py3-none-any.whl

[rtibbles]

DMG build failure appears to be unrelated to these changes. Tracked separately: https://github.com/learningequality/kolibri-app/issues/105

[rtibbles]

That's quite a high percentage!

[vkWeb]

That's quite a high percentage!

Very high indeed sir!

[rtibbles]

Looks like I've messed up the argument handling for the diff file stats too, will fix!

[rtibbles]

Have fixed the import diff stats issue, upgrading now works properly.

I did see a frontend issue during the upgrade process, but the backend experienced no errors. Not sure if the issue is also extant on develop, as I've not touched the frontend here.

Progress updating is now working properly again - I had simply forgotten to set the total progress at the start!

[bjester]

Upgrading a channel, I got the following frontend error:

vue.runtime.esm.js:1897 TypeError: Cannot read properties of undefined (reading 'id')
    at NewChannelVersionPage.vue:237:1
logError @ vue.runtime.esm.js:1897

Also, does this mean a resource was a duplicate or that it failed to import?

[rtibbles]

Upgrading a channel, I got the following frontend error:

I got the same issue here, it seemed like a frontend only issue, but can double check this and fix (I think it might be pre-existing in develop but worth fixing here).

Also, does this mean a resource was a duplicate or that it failed to import?

Yeah, this is a pre-existing issue with resource counting being deduped in one place and not in another. I'll fix it in this PR in case it doesn't get fixed in 0.15.

[rtibbles]

Updated - add a from_manifest class method to handle the manifest construction and make the keyword arguments explicit.

Cleaned up various refactor induced bugs in the channel update workflow. Should be good to go!