Incremental changelog doesn't remove *_content_type column on removal of an AnyBlob field

[OmarHawk]

Overview of the issue

We do have a schema, where there is a column with type AnyBlob in there. The application was initially generated using the --incremental-changelog flag.

The Entity looks initially like this (also can be reproduced with larger schemas of course):

entity DemoRouter {
   contentString AnyBlob
}

This initial schema looks like this in the liquibase changeset:

    <changeSet id="20220119144011-1" author="jhipster">
        <createTable tableName="demo_router">
            <column name="id" type="bigint" autoIncrement="true">
                <constraints primaryKey="true" nullable="false"/>
            </column>
            <column name="content_string" type="longblob">
                <constraints nullable="true" />
            </column>
            <column name="content_string_content_type" type="varchar(255)">
                <constraints nullable="true" />
            </column>
            <!-- jhipster-needle-liquibase-add-column - JHipster will add columns here -->
        </createTable>
    </changeSet>

Please note the column content_string_content_type.

After some time, we do want to get rid of the contentString field and (possibly) have another (unrelated) field instead, so the schema now looks like this:

entity DemoRouter {
   anotherColumn String
}

When we now run the jhipster jdl demo.jdl command, it produces a new liquibase changeset, that looks like this:

   <!--
        Added columns to the entity DemoRouter.
    -->
    <changeSet id="20220119144054-1-add-columns" author="jhipster">
        <addColumn tableName="demo_router">
            <column name="another_column" type="varchar(255)"/>
        </addColumn>
    </changeSet>

    <!--
        Dropped columns to the entity DemoRouter.
    -->
    <changeSet id="20220119144054-1-drop-columns" author="jhipster">
        <dropColumn tableName="demo_router">
            <column name="content_string"/>
        </dropColumn>
    </changeSet>

So it drops the Blob column content_string itself and adds a varchar column another_column instead. But it doesn't get rid of the helper column content_string_content_type for the Blob, leaving remains in the database that are not needed anymore and lead to a column that is not referenced anywhere anymore.

Motivation for or Use Case

Database now contains a column which is not under any control anymore.

Reproduce the error

Create an application with a AnyBlob column in one of the entities. Remove the AnyBlob column and let a changelog be generated.

Related issues

Haven't found any.

Suggest a Fix

Whereever the changesets are generated consider also special data types where one entity field produces more than one column (like AnyBlob).

JHipster Version(s)

7.5.0

JHipster configuration

C:\Users\jwedding\git\markant\demo2>jhipster info INFO! Using JHipster version installed locally in current project's node_modules Welcome to the JHipster Information Sub-Generator

JHipster Version(s)

[email protected] C:\Users\jwedding\git\markant\demo2
`-- [email protected]

JHipster configuration, a .yo-rc.json file generated in the root folder

.yo-rc.json file

{
  "generator-jhipster": {
    "incrementalChangelog": true,
    "applicationType": "monolith",
    "baseName": "demo2",
    "jhipsterVersion": "7.5.0",
    "skipUserManagement": false,
    "skipCheckLengthOfIdentifier": false,
    "skipFakeData": false,
    "jhiPrefix": "jhi",
    "entitySuffix": "",
    "dtoSuffix": "DTO",
    "testFrameworks": [],
    "blueprints": [],
    "otherModules": [],
    "pages": [],
    "creationTimestamp": 1642603036325,
    "serviceDiscoveryType": "no",
    "reactive": false,
    "authenticationType": "session",
    "packageName": "com.mycompany.myapp",
    "serverPort": "8080",
    "cacheProvider": "no",
    "enableHibernateCache": false,
    "databaseType": "sql",
    "devDatabaseType": "mysql",
    "prodDatabaseType": "mysql",
    "buildTool": "maven",
    "serverSideOptions": [],
    "websocket": false,
    "searchEngine": false,
    "messageBroker": false,
    "enableSwaggerCodegen": false,
    "clientFramework": "no",
    "skipClient": true,
    "enableTranslation": true,
    "nativeLanguage": "de",
    "rememberMeKey": "YourJWTSecretKeyWasReplacedByThisMeaninglessTextByTheJHipsterInfoCommandForObviousSecurityReasons",
    "devServerPort": 9060,
    "clientPackageManager": "npm",
    "clientTheme": "none",
    "clientThemeVariant": "",
    "withAdminUi": true,
    "languages": ["de", "en"],
    "enableGradleEnterprise": false,
    "entities": ["DemoRouter"],
    "lastLiquibaseTimestamp": 1642603254000
  }
}

JDL for the Entity configuration(s) entityName.json files generated in the .jhipster directory

JDL entity definitions

entity DemoRouter {
  anotherColumn String
}
paginate DemoRouter with pagination
service DemoRouter with serviceImpl

Environment and Tools

openjdk version "11.0.12" 2021-07-20 LTS OpenJDK Runtime Environment 18.9 (build 11.0.12+7-LTS) OpenJDK 64-Bit Server VM 18.9 (build 11.0.12+7-LTS, mixed mode)

git version 2.34.1.windows.1

node: v14.18.2

npm: 6.14.15

Docker version 20.10.8, build 3967b7d

docker-compose version 1.29.2, build 5becea4c

Entity configuration(s) entityName.json files generated in the .jhipster directory

Final one after steps have been performed:

{
  "name": "DemoRouter",
  "fields": [
    {
      "fieldName": "anotherColumn",
      "fieldType": "String"
    }
  ],
  "relationships": [],
  "entityTableName": "demo_router",
  "dto": "no",
  "pagination": "pagination",
  "service": "serviceImpl",
  "jpaMetamodelFiltering": false,
  "fluentMethods": true,
  "readOnly": false,
  "embedded": false,
  "applications": "*",
  "changelogDate": "20220119144011",
  "incrementalChangelog": true
}

Browsers and Operating System

Windows 10

• [x] Checking this box is mandatory (this is just to show you read everything)

[mraible]

@OmarHawk Are you interested in trying to fix this if I assign a bug bounty to it?

[OmarHawk]

@mraible sure, but I will only come to this somewhen next week :-)

[mshima]

I may try to convert the content_type field to an actual field for v8. So, instead of implementing at templates, we should inject the field to fields list. Templates won’t need to add all those if blob then generate the content_type field anymore.

[DanielFran]

@OmarHawk Are you available to provide a PR?

[OmarHawk]

well, If I'd have more time, definetly. Will check it out somwhen this week maybe, but can't promise anything. The necessary change is minimal tbh.