CDS Videos: Upgrade dependencies

[zzacharo]

Code

• Development branch
• Needed fixes (Both changes are installed locally when you run ./scripts/bootstrap_upgrade)
  • invenio admin
  • invenio-sequencegenerator

Known issues / missing parts

• integrate invenio-stats
• deletion of project gives a "PIDDeletedError". The project and video is correctly deleted but the error seems to come from an async indexing that tries to resolve the deleted video
• deposit form: various problems with rendering the angular schema form fields. Overridden templates are not loaded correctly
• upgrade the docker image to alma 9
• invenio-deposit was moved in a module inside the main codebase for simplicity. Once things are correctly working, we could simplify it even more and integrate the code with the previously existing deposit local module.
• missing os-v2 invenio_opendefinition.mappings warning in the console

Done parts

• moved record indexing to opensearch (mappings update, code relevant changes)
• all dependencies were upgraded
• moved to webpack bundling

Before deployment

• Ensure SECURITY_PASSWORD_SINGLE_HASH = True, needed for the existing passwords and token to continue working
• DB alembic upgrade i.e. invenio alembic upgrade
• Keywords update as the suggesting mechanism changed due to opensearch mappings change (suggest endpoint doesn't accept payload anymore)

from sqlalchemy.orm.exc import NoResultFound

from cds.modules.records.api import Keyword
from invenio_pidstore.models import PersistentIdentifier, PIDStatus
from invenio_db import db


def get_all_pids_by(pid_type):
    """Get all PIDs for the given type.

    :param pid_type: String representing the PID type value, for example 'recid'.
    """
    pids = PersistentIdentifier.query.filter(PersistentIdentifier.pid_type == pid_type).filter(
        PersistentIdentifier.status == PIDStatus.REGISTERED).yield_per(100)
    return pids


def get_record_by_pid(pid):
    """Fetch and return the published record given its PID.

    :param pid: String representing the PID value.
    """
    try:
        return Keyword.get_record(pid.object_uuid)
    except NoResultFound:
        return None


kwds = get_all_pids_by("kwid")

for pid in kwds:
    rec = get_record_by_pid(pid)
    if rec is not None:
        rec["suggest_name"].pop("payload", None)
        rec.commit()
db.session.commit()

• Reindexing of all content

cds index reindex -t recid
cds index reindex -t depid
cds index reindex -t catid
cds index reindex -t kwid
cds index run

Local setup / development

Pre-requisites

• Node v18
• Python 3.9

Setup

# setup
./scripts/bootstrap_upgrade
./scripts/setup-instance.sh

# web server
./scripts/server

# celery
./scripts/celery

# assets watch
./scripts/assets-watch

Upgrade notes

• (Check if needed) Flask-limiter is breaking invenio-records-ui partial views. To find the views in videos run in PaaS

import functools
[v for v in app.view_functions.values() if isinstance(v, functools.partial)]
[functools.partial(<function record_view at 0x7efc7565cd08>, resolver=<invenio_pidstore.resolver.Resolver object at 0x7efc756644e0>, template='cds_deposit/edit.html', permission_factory=None, view_method=<function default_view_method at 0x7efc7565cd90>),
 functools.partial(<function record_view at 0x7efc7565cd08>, resolver=<invenio_pidstore.resolver.Resolver object at 0x7efc75664208>, template='cds_deposit/edit.html', permission_factory=None, view_method=<function default_view_method at 0x7efc7565cd90>),
 functools.partial(<function record_view at 0x7efc7565cd08>, resolver=<invenio_pidstore.resolver.Resolver object at 0x7efc756640f0>, template='cds_deposit/edit.html', permission_factory=None, view_method=<function project_view at 0x7efc755f9268>),
 functools.partial(<function record_view at 0x7efc7565cd08>, resolver=<invenio_pidstore.resolver.Resolver object at 0x7efc75447ac8>, template='cds_records/record_detail.html', permission_factory=None, view_method=<function default_view_method at 0x7efc7565cd90>),
 functools.partial(<function record_view at 0x7efc7565cd08>, resolver=<invenio_pidstore.resolver.Resolver object at 0x7efc75447940>, template='cds_records/record_stats.html', permission_factory=None, view_method=<function stats_recid at 0x7efc754311e0>),
 functools.partial(<function record_view at 0x7efc7565cd08>, resolver=<invenio_pidstore.resolver.Resolver object at 0x7efc75447898>, template='invenio_records_ui/detail.html', permission_factory=None, view_method=<function preview_recid at 0x7efc754316a8>),
 functools.partial(<function record_view at 0x7efc7565cd08>, resolver=<invenio_pidstore.resolver.Resolver object at 0x7efc75447be0>, template='invenio_records_ui/detail.html', permission_factory=None, view_method=<function preview_recid_embed at 0x7efc754318c8>),
 functools.partial(<function record_view at 0x7efc7565cd08>, resolver=<invenio_pidstore.resolver.Resolver object at 0x7efc75447c18>, template='invenio_records_ui/detail.html', permission_factory=None, view_method=<function preview_recid_embed at 0x7efc754318c8>),
 functools.partial(<function record_view at 0x7efc7565cd08>, resolver=<invenio_pidstore.resolver.Resolver object at 0x7efc75447978>, template='invenio_records_ui/detail.html', permission_factory=None, view_method=<function file_download_ui at 0x7efc7e705d90>),
 functools.partial(<function record_view at 0x7efc7565cd08>, resolver=<invenio_pidstore.resolver.Resolver object at 0x7efc75430898>, template='invenio_records_ui/detail.html', permission_factory=None, view_method=<function preview_depid at 0x7efc75431840>),
 functools.partial(<function record_view at 0x7efc7565cd08>, resolver=<invenio_pidstore.resolver.Resolver object at 0x7efc75430b00>, template='invenio_records_ui/detail.html', permission_factory=None, view_method=<function preview_depid at 0x7efc75431840>),
 functools.partial(<function record_view at 0x7efc7565cd08>, resolver=<invenio_pidstore.resolver.Resolver object at 0x7efc75430ba8>, template='cds_records/record_export.html', permission_factory=None, view_method=<function records_ui_export at 0x7efc75431510>),
 functools.partial(<function record_view at 0x7efc7565cd08>, resolver=<invenio_pidstore.resolver.Resolver object at 0x7efc75430be0>, template='invenio_records_ui/detail.html', permission_factory=<function record_delete_permission_factory at 0x7efc7bd2ed08>, view_method=<function records_ui_delete at 0x7efc75431598>)]

• [ ] Check the progress/solution in the opendata repo
• [ ] check https://github.com/inveniosoftware/invenio-records-ui/pull/101
• [ ]