createIndex if 'NOT indexExists' fails (version 4.15)

[jmayday]

Demo project: https://github.com/jmayday/liquibase-demo Liquibase version: 4.15.0 #4001 built at 2022-08-05 16:17+0000

I'm adding Liquibase integration to project and I would like to prepare a change set creating indexes if they are missing only. Someone already asked about indexExists precondition here https://github.com/liquibase/liquibase-mongodb/issues/69 but there was no answer.

This is change set without any precondition and it fails with error below (which is all correct, as there is already index with such name):

<changeSet id="changeset_20220822_1250_1" author="jmayday">
    <ext:createIndex collectionName="testCollection">
        <ext:keys>
            { userId: 1, type: 1}
        </ext:keys>
        <ext:options>
            { unique: false, name: "userId_1" }
        </ext:options>
    </ext:createIndex>
</changeSet>

The error message (again - all is fine, index is already existing so we can't 2nd one with same name):

Caused by: com.mongodb.MongoCommandException: Command failed with error 86 (IndexKeySpecsConflict): 'Index must have unique name.The existing index: { v: 2, key: { userId: 1 }, name: "userId_1", ns: "dev1.testCollection" } has the same name as the requested index: { v: 2, key: { userId: 1, type: 1 }, name: "userId_1", ns: "dev1.testCollection" }' on server localhost:27017. The full response is {"ok": 0.0, "errmsg": "Index must have unique name.The existing index: { v: 2, key: { userId: 1 }, name: \"userId_1\", ns: \"dev1.testCollection\" } has the same name as the requested index: { v: 2, key: { userId: 1, type: 1 }, name: \"userId_1\", ns: \"dev1.testCollection\" }", "code": 86, "codeName": "IndexKeySpecsConflict"}
	at com.mongodb.internal.connection.ProtocolHelper.getCommandFailureException(ProtocolHelper.java:198)

But if I extend the changeSet with preConditions, then an exception is being thrown:

<changeSet id="changeset_20220822_1250_1" author="jmayday">
    <preConditions onFail="MARK_RAN">
        <not>
            <indexExists indexName="userId_1"/>
        </not>
    </preConditions>
    <ext:createIndex collectionName="testCollection">
        <ext:keys>
            { userId: 1, type: 1}
        </ext:keys>
        <ext:options>
            { unique: false, name: "userId_1" }
        </ext:options>
    </ext:createIndex>
</changeSet>

The error:

Exception in thread "main" liquibase.exception.LiquibaseException: liquibase.exception.MigrationFailedException: Migration failed for changeset changesets/changeset_20220819_1428.xml::changeset_20220822_1250_1::jmayday:
     Reason: 
          changelog.xml : Index Exists Precondition: userId_1 : class liquibase.ext.mongodb.database.MongoConnection cannot be cast to class liquibase.database.jvm.JdbcConnection (liquibase.ext.mongodb.database.MongoConnection and liquibase.database.jvm.JdbcConnection are in unnamed module of loader 'app')

Dependency tree:

[INFO] com.myproject:liquibase:jar:1.0.0-SNAPSHOT
[INFO] +- org.liquibase.ext:liquibase-mongodb:jar:4.15.0:compile
[INFO] |  +- org.liquibase:liquibase-core:jar:4.15.0:compile
[INFO] |  |  +- javax.xml.bind:jaxb-api:jar:2.3.1:compile
[INFO] |  |  |  \- javax.activation:javax.activation-api:jar:1.2.0:compile
[INFO] |  |  +- org.yaml:snakeyaml:jar:1.27:compile
[INFO] |  |  \- com.opencsv:opencsv:jar:5.6:compile
[INFO] |  |     +- org.apache.commons:commons-lang3:jar:3.12.0:compile
[INFO] |  |     +- org.apache.commons:commons-text:jar:1.9:compile
[INFO] |  |     \- org.apache.commons:commons-collections4:jar:4.4:compile
[INFO] |  +- org.mongodb:mongodb-driver-sync:jar:4.6.1:compile
[INFO] |  +- org.mongodb:mongodb-driver-core:jar:4.6.1:compile
[INFO] |  |  \- org.mongodb:bson-record-codec:jar:4.6.1:runtime
[INFO] |  +- org.mongodb:bson:jar:4.6.1:compile
[INFO] |  +- com.fasterxml.jackson.core:jackson-databind:jar:2.13.3:compile
[INFO] |  |  +- com.fasterxml.jackson.core:jackson-annotations:jar:2.13.3:compile
[INFO] |  |  \- com.fasterxml.jackson.core:jackson-core:jar:2.13.3:compile
[INFO] |  \- org.projectlombok:lombok:jar:1.18.22:compile
[INFO] +- commons-cli:commons-cli:jar:1.5.0:compile
[INFO] \- ch.qos.logback:logback-classic:jar:1.2.11:compile
[INFO]    +- ch.qos.logback:logback-core:jar:1.2.11:compile
[INFO]    \- org.slf4j:slf4j-api:jar:1.7.30:compile

When debugging I find method liquibase.snapshot.JdbcDatabaseSnapshot#getMetaDataFromCache to do casting which throws exception because getDatabase().getConnection() is a MongoConnection:

public CachingDatabaseMetaData getMetaDataFromCache() throws SQLException {
    if (cachingDatabaseMetaData == null) {
        DatabaseMetaData databaseMetaData = null;
        if (getDatabase().getConnection() != null) {
            databaseMetaData = ((JdbcConnection) getDatabase().getConnection()).getUnderlyingConnection().getMetaData();
        }

        cachingDatabaseMetaData = new CachingDatabaseMetaData(this.getDatabase(), databaseMetaData);
    }
    return cachingDatabaseMetaData;
}

Whole stacktrace below:

Exception in thread "main" liquibase.exception.LiquibaseException: liquibase.exception.MigrationFailedException: Migration failed for changeset changesets/changeset_20220819_1428.xml::changeset_20220822_1250_1::jmayday:
     Reason: 
          changelog.xml : Index Exists Precondition: userId_1 : class liquibase.ext.mongodb.database.MongoConnection cannot be cast to class liquibase.database.jvm.JdbcConnection (liquibase.ext.mongodb.database.MongoConnection and liquibase.database.jvm.JdbcConnection are in unnamed module of loader 'app')

	at liquibase.changelog.ChangeLogIterator.run(ChangeLogIterator.java:126)
	at liquibase.Liquibase.lambda$null$0(Liquibase.java:263)
	at liquibase.Scope.lambda$child$0(Scope.java:180)
	at liquibase.Scope.child(Scope.java:189)
	at liquibase.Scope.child(Scope.java:179)
	at liquibase.Scope.child(Scope.java:158)
	at liquibase.Scope.child(Scope.java:243)
	at liquibase.Liquibase.lambda$update$1(Liquibase.java:262)
	at liquibase.Scope.lambda$child$0(Scope.java:180)
	at liquibase.Scope.child(Scope.java:189)
	at liquibase.Scope.child(Scope.java:179)
	at liquibase.Scope.child(Scope.java:158)
	at liquibase.Liquibase.runInScope(Liquibase.java:2414)
	at liquibase.Liquibase.update(Liquibase.java:209)
	at liquibase.Liquibase.update(Liquibase.java:195)
	at liquibase.Liquibase.update(Liquibase.java:191)
	at liquibase.Liquibase.update(Liquibase.java:183)
	at com.myproject.Liqui.main(Liqui.java:62)
Caused by: liquibase.exception.MigrationFailedException: Migration failed for changeset changesets/changeset_20220819_1428.xml::changeset_20220822_1250_1::jmayday:
     Reason: 
          changelog.xml : Index Exists Precondition: userId_1 : class liquibase.ext.mongodb.database.MongoConnection cannot be cast to class liquibase.database.jvm.JdbcConnection (liquibase.ext.mongodb.database.MongoConnection and liquibase.database.jvm.JdbcConnection are in unnamed module of loader 'app')

	at liquibase.changelog.ChangeSet.execute(ChangeSet.java:632)
	at liquibase.changelog.visitor.UpdateVisitor.visit(UpdateVisitor.java:56)
	at liquibase.changelog.ChangeLogIterator$2.lambda$null$0(ChangeLogIterator.java:113)
	at liquibase.Scope.lambda$child$0(Scope.java:180)
	at liquibase.Scope.child(Scope.java:189)
	at liquibase.Scope.child(Scope.java:179)
	at liquibase.Scope.child(Scope.java:158)
	at liquibase.changelog.ChangeLogIterator$2.lambda$run$1(ChangeLogIterator.java:112)
	at liquibase.Scope.lambda$child$0(Scope.java:180)
	at liquibase.Scope.child(Scope.java:189)
	at liquibase.Scope.child(Scope.java:179)
	at liquibase.Scope.child(Scope.java:158)
	at liquibase.Scope.child(Scope.java:243)
	at liquibase.changelog.ChangeLogIterator$2.run(ChangeLogIterator.java:93)
	at liquibase.Scope.lambda$child$0(Scope.java:180)
	at liquibase.Scope.child(Scope.java:189)
	at liquibase.Scope.child(Scope.java:179)
	at liquibase.Scope.child(Scope.java:158)
	at liquibase.Scope.child(Scope.java:243)
	at liquibase.Scope.child(Scope.java:247)
	at liquibase.changelog.ChangeLogIterator.run(ChangeLogIterator.java:65)
	... 17 more
Caused by: liquibase.exception.PreconditionErrorException: Precondition Error
	at liquibase.precondition.core.IndexExistsPrecondition.check(IndexExistsPrecondition.java:123)
	at liquibase.precondition.core.NotPrecondition.check(NotPrecondition.java:35)
	at liquibase.precondition.core.AndPrecondition.check(AndPrecondition.java:40)
	at liquibase.precondition.core.PreconditionContainer.check(PreconditionContainer.java:213)
	at liquibase.changelog.ChangeSet.execute(ChangeSet.java:589)
	... 37 more
Caused by: liquibase.exception.DatabaseException: java.lang.ClassCastException: class liquibase.ext.mongodb.database.MongoConnection cannot be cast to class liquibase.database.jvm.JdbcConnection (liquibase.ext.mongodb.database.MongoConnection and liquibase.database.jvm.JdbcConnection are in unnamed module of loader 'app')
	at liquibase.snapshot.jvm.IndexSnapshotGenerator.snapshotObject(IndexSnapshotGenerator.java:304)
	at liquibase.snapshot.jvm.JdbcSnapshotGenerator.snapshot(JdbcSnapshotGenerator.java:66)
	at liquibase.snapshot.SnapshotGeneratorChain.snapshot(SnapshotGeneratorChain.java:49)
	at liquibase.snapshot.DatabaseSnapshot.include(DatabaseSnapshot.java:312)
	at liquibase.snapshot.DatabaseSnapshot.init(DatabaseSnapshot.java:105)
	at liquibase.snapshot.DatabaseSnapshot.<init>(DatabaseSnapshot.java:58)
	at liquibase.snapshot.JdbcDatabaseSnapshot.<init>(JdbcDatabaseSnapshot.java:34)
	at liquibase.snapshot.SnapshotGeneratorFactory.createSnapshot(SnapshotGeneratorFactory.java:215)
	at liquibase.snapshot.SnapshotGeneratorFactory.createSnapshot(SnapshotGeneratorFactory.java:244)
	at liquibase.snapshot.SnapshotGeneratorFactory.has(SnapshotGeneratorFactory.java:133)
	at liquibase.precondition.core.IndexExistsPrecondition.check(IndexExistsPrecondition.java:103)
	... 41 more
Caused by: java.lang.ClassCastException: class liquibase.ext.mongodb.database.MongoConnection cannot be cast to class liquibase.database.jvm.JdbcConnection (liquibase.ext.mongodb.database.MongoConnection and liquibase.database.jvm.JdbcConnection are in unnamed module of loader 'app')
	at liquibase.snapshot.JdbcDatabaseSnapshot.getMetaDataFromCache(JdbcDatabaseSnapshot.java:45)
	at liquibase.snapshot.jvm.IndexSnapshotGenerator.snapshotObject(IndexSnapshotGenerator.java:159)
	... 51 more

[r-michal-ah]

hello, any update on it?

[kb-mendozaACN]

Can we have this fixed pls?

[jmayday]

@r-michal-ah @kb-mendozaACN I don't think it's worth waiting. I've started using Mongock, maybe it will be good choice for you guys.

[kb-mendozaACN]

@jmayday unfortunately, we can not introduce more tools. have you tried creating via mongodb:runCommand? I wanted to try it as well, but checking here first, just in case someone already tried and failed too.

[jmayday]

Sorry, I don't understand. What you mean exactly?

[kb-mendozaACN]

@jmayday using this https://github.com/liquibase/liquibase-mongodb/blob/main/src/test/resources/liquibase/ext/changelog.run-command.test.xml but for createIndex command (https://www.mongodb.com/docs/v3.4/reference/command/createIndexes/)?

[jmayday]

I think I tried. But I don't remember exactly, as it was year ago :). I think I exhausted options to use liquibase with Mongo, and there was no support at all (as we see in this thread), so I started checking alternatives and ended up using Mongock.

[pcalouche]

I would love to see this implemented to. The workaround I found was not fail the change set if there was an error. Better than nothing.

<?xml version="1.0" encoding="UTF-8"?>
<databaseChangeLog
  xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xmlns:ext="http://www.liquibase.org/xml/ns/dbchangelog-ext"
  xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog
        https://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-latest.xsd
        http://www.liquibase.org/xml/ns/dbchangelog-ext https://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-ext.xsd">

  <!--
    Mongo Liquibase does not have a precondition to check if an index exists, so set
    failOnError to be false. See https://github.com/liquibase/liquibase-mongodb/issues/283.
  -->
  <changeSet id="4" author="psc" failOnError="false">
    <ext:dropIndex collectionName="testers">
      <ext:keys>{firstName: 1}</ext:keys>
    </ext:dropIndex>
    <rollback>
      <ext:createIndex collectionName="testers">
        <ext:keys>{firstName: 1}</ext:keys>
        <ext:options>{name: "firstName_asc"}</ext:options>
      </ext:createIndex>
    </rollback>
    <comment>
      Drops the index on the firstName field in the testers collection
    </comment>
  </changeSet>

</databaseChangeLog>