Missing database migration tasks when migrating from <=5.1.6

[yolabingo]

Describe the bug After migrating a site from <=5.1.6 to current LTS release, the database is missing several columns. Presumably there are missing database migration tasks.

To Reproduce Steps to reproduce the behavior:

1. do a clean install of 5.1.6 and dump the database
2. using the 5.1.6 database, start dotcms on current 5.3.8, 21.06 or 22.03 to run the migrations
3. observe sql errors in the dotcms log due to missing database columns e.g.

ERROR business.RelationshipFactoryImpl - Error getting relationships with typeValue: news-comments
com.dotmarketing.exception.DotDataException: ERROR: column "mod_date" does not exist
  Position: 179{
  "SQL": ["select inode, parent_structure_inode, child_structure_inode, parent_relation_name, child_relation_name, relation_type_value, cardinality, parent_required, child_required, fixed, mod_date from relationship where lower(relation_type_value) = ?"],

Current Workaround Manually applying the following changes to the 5.1.6 database prior to running dotCMS on newer versions seems to solve the issues.

Before running 5.3.8 migrations, run:

ALTER TABLE dot_cluster ADD cluster_salt CHARACTER VARYING(256);

Before running 21.06 migrations, run:

ALTER TABLE dot_cluster ADD cluster_salt CHARACTER VARYING(256);
ALTER TABLE sitelic ADD startup_time BIGINT;

Before running 22.03 migrations, run:

ALTER TABLE dot_cluster ADD cluster_salt CHARACTER VARYING(256);
ALTER TABLE sitelic ADD startup_time BIGINT;
ALTER TABLE structure ADD icon VARCHAR(255);
ALTER TABLE structure ADD sort_order INTEGER;
ALTER TABLE relationship ADD mod_date TIMESTAMP WITH TIME ZONE;
ALTER TABLE company ADD default_language_id BIGINT;

Tested on Postgres only

clean dotcms 5.1.6 database

[dsilvam]

The upgrade task https://github.com/dotCMS/core/blob/master/dotCMS/src/main/java/com/dotmarketing/startup/runonce/Task05380ChangeContainerPathToAbsolute.java#L22 needs to be modified to avoid making API calls. At some point these API calls hit the DB and they don't have yet the expected alterations so the queries can run, hence having to run alterations manually so the upgrade can succeed.

[dsilvam]

Passed Internal QA: Upgraded from 5.1.6 to master (22.08) without failures. The task Task05380ChangeContainerPathToAbsoluteTest was the one causing the failures, so since it was changed its functionality needs to be re-tested.

To re-test the Upgrade Task functionality do this:

1. After importing the dump from 5.1.6 run the following query:

UPDATE template
set drawed_body = REPLACE(drawed_body, 'a050073a-a31e-4aab-9307-86bfb248096a', '/application/containers/large-column/')
WHERE drawed_body  LIKE '%a050073a-a31e-4aab-9307-86bfb248096a%' 

It needs to update 20 rows.

2. Then run this query:

select *
from template
WHERE drawed_body LIKE '%//demo.dotcms.com/application/containers/large-column/%'

It needs to show 0 results.

3. Run the upgrade and make sure the task 5380 ran by looking at the logs. Then run this query again:

select *
from template
WHERE drawed_body LIKE '%//demo.dotcms.com/application/containers/large-column/%'

It needs to return 20 rows.

[erickgonzalez]

Not apply to 5.3.8 LTS since it's failing because of the StartupTask Cluster.

If I'm not wrong before upgrading from < 5.3.8 we will need to run these queries:

ALTER TABLE dot_cluster ADD cluster_salt CHARACTER VARYING(256);
ALTER TABLE sitelic ADD startup_time BIGINT;

[bryanboza]

Fixed, tested upgrading from 5.1.6 to release-22.09 branch using the provided steps