liquibase
liquibase copied to clipboard
liquibase
Changeset execution information properties
Impact
- [ ] Bug fix (non-breaking change which fixes expected existing functionality)
- [X] Enhancement/New feature (adds functionality without impacting existing logic)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
Description
Fixes #5578
This PR adds the ability to reference the following properties in changelogs:
- ~~LIQUIBASE_EXECUTION_DEPLOYMENT_ID~~ - Removed from this PR and has been moved to #5653 due to complications discussed in thread below
- LIQUIBASE_EXECUTION_CHANGELOG_FILE
- LIQUIBASE_EXECUTION_CHANGESET_ID
- LIQUIBASE_EXECUTION_CHANGESET_AUTHOR
Allowing for developers to be able to access the above information (for example logging liquibase install information).
Things to be aware of
I added two directories to the .gitignore file because they were created when I ran mvn clean install when there was no differences between my branch and the master branch. If this is unwanted, I can easily remove the .gitignore entries... but I figured it made sense to add them. Let me know what to do in this regards.
Things to worry about
Additional Context
Hey @filipelautert! Hope you have been well!
I created this PR in draft because I'd like some help/opinions here. I've got the majority of this working, but can't figure out how to get the correct DEPLOYMENT_ID.
It appears that the properties are all replaced in changelog files before a lock is accquired and the update action is actually run. When the update run is started, accquiring the change lock resets the deployment ID (as explained by this comment.
So my question is... any thoughts on how I could achieve getting the deployment ID so it is available to the changes?
I'm not hard set on the usage of the properties for this... that is just the first thing that came to mind as that substitution is already built in to Liquibase. Not sure if you or your team mates could think of another mechanism that would be better suited?
Thanks!
So my question is... any thoughts on how I could achieve getting the deployment ID so it is available to the changes?
Hey @jasonlyle88 ! Could you provide an example changelog were you would use the deployment id?
But you are right.. we reset the deployment id with each reset. And that's like this because it didn't matter until now as the deployment id is the same for all the operations executed in a single update . I'm thinking what would be the implications of creating one deployment id and never resetting it, especially for multitenant environments. In this case you would have the same id in different databases.. maybe something else would change? Or maybe flows would be affected too? Also we can't remove the resets as they are important in multithread environments.
OR maybe it would be easier to parse the deployment id parameter after the code comment you mentioned .
Hey,
Here is an example of how I was thinking this could be used
<?xml version="1.0" encoding="UTF-8"?>
<databaseChangeLog
xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:ora="http://www.oracle.com/xml/ns/dbchangelog-ext"
xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-4.17.xsd"
>
<changeSet
id="example"
author="jlyle"
>
<sql endDelimiter="/">
begin
-- Do installation stuff
...
exception when others
--ERROR HANDLER
logger.log_error(
p_text => 'Error with liquibase run',
p_additional_info => t_info_tab(
t_info_row('Error Code', sqlcode),
t_info_row('Error Message', sqlerrm),
t_info_row('Deployment ID', '${LIQUIBASE_EXECUTION_DEPLOYMENT_ID}'),
t_info_row('Changelog File', '${LIQUIBASE_EXECUTION_CHANGELOG_FILE}'),
t_info_row('Changeset ID', '${LIQUIBASE_EXECUTION_CHANGESET_ID}'),
t_info_row('Changeset Author', '${LIQUIBASE_EXECUTION_CHANGESET_AUTHOR}')
)
);
end;
/
</sql>
</changeSet>
</databaseChangeLog>
I agree only having a singular deployment ID could be problematic. Technically, 10 liquibase updates could be executed at almost the same time and end up with the same deployment ID (since it is time based). But only the first one to get the lock executes and the deployment ID is recalculated at that time. This would distinguish each update attempt because its deployment ID time is based on when it got the lock, not when it was executed. Basically, I agree that the resets are important.
I would be interested in how you think the deployment ID could be parsed after the lock is obtained because that would definitely solve the problem. Are you thinking move all parameter replacements to after the database lock is established? Or somehow just doing it for this one special case?
I suppose we could have a singular deployment ID that does not get reset if we make it more unique. To that end, I did a quick test with the java UUID:
import java.util.UUID;
import java.math.BigInteger;
class UUIDToInt {
// Shows that the UUID theoretically can span anything from 1 digit to 39 digits
public static void main(String[] args) {
UUID uuid;
String uuidString;
BigInteger bi;
String integerString;
int shortestLength = 99999999;
int integerLength;
for (int i = 0; i < 30; i++) {
uuid = UUID.randomUUID();
uuidString = uuid.toString().replaceAll( "-" , "" );
if (i == 0) {
// Test smallest possible number
uuidString="00000000000000000000000000000000";
}
else if (i == 1) {
// Test largest possible number
uuidString="ffffffffffffffffffffffffffffffff";
}
bi = new BigInteger( uuidString , 16 );
integerString = bi.toString() ;
integerLength = integerString.length();
if (i > 1 && integerLength < shortestLength) {
shortestLength = integerLength;
}
System.out.println(integerLength + ": " + integerString);
}
System.out.println("Shortest generated length: " + shortestLength);
}
}
This shows that theoretically a generated integer from UUID can be anywhere from 1-39 digits long. Although, practically it will most likely be 36+ digits long from my testing.
If it is acceptable to convert the deployment ID to a UUID string (always 36 characters long, 32 without dashes) or a UUID numeric representation (0-39 characters long), that would allow a single deployment ID to be generated and not need to be regenerated ever since it is unique.
For reference, the deployment ID column is currently a character column of length 10.
One difference in the UUID based number approach that should be noted is that if N values are generated, the Nth + 1 value is not guaranteed to be greater than the Nth value.
Right now, since the DeploymentID is time based, the next generation is always a greater (assuming that it is generated at a different time, which is guaranteed by the database lock)
Hey,
Here is an example of how I was thinking this could be used
<?xml version="1.0" encoding="UTF-8"?> <databaseChangeLog xmlns="http://www.liquibase.org/xml/ns/dbchangelog" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:ora="http://www.oracle.com/xml/ns/dbchangelog-ext" xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-4.17.xsd" > <changeSet id="example" author="jlyle" > <sql endDelimiter="/"> begin -- Do installation stuff ... exception when others --ERROR HANDLER logger.log_error( p_text => 'Error with liquibase run', p_additional_info => t_info_tab( t_info_row('Error Code', sqlcode), t_info_row('Error Message', sqlerrm), t_info_row('Deployment ID', '${LIQUIBASE_EXECUTION_DEPLOYMENT_ID}'), t_info_row('Changelog File', '${LIQUIBASE_EXECUTION_CHANGELOG_FILE}'), t_info_row('Changeset ID', '${LIQUIBASE_EXECUTION_CHANGESET_ID}'), t_info_row('Changeset Author', '${LIQUIBASE_EXECUTION_CHANGESET_AUTHOR}') ) ); end; / </sql> </changeSet> </databaseChangeLog>I agree only having a singular deployment ID could be problematic. Technically, 10 liquibase updates could be executed at almost the same time and end up with the same deployment ID (since it is time based). But only the first one to get the lock executes and the deployment ID is recalculated at that time. This would distinguish each update attempt because its deployment ID time is based on when it got the lock, not when it was executed. Basically, I agree that the resets are important.
I would be interested in how you think the deployment ID could be parsed after the lock is obtained because that would definitely solve the problem. Are you thinking move all parameter replacements to after the database lock is established? Or somehow just doing it for this one special case?
Right, that's the idea! I discussed with some people here and we agreed that if it is possible to move the replacement code inside AbstractUpdateCommandStep it would be great (I'm not sure if it would work). Otherwise keep the parameter replacement where is it, but if it finds LIQUIBASE_EXECUTION_DEPLOYMENT_ID then flag it to be replaced after the last lock is acquired.
What do you think?
I'm absolutely up for either of those items. I haven't dug to much into the AbstractUpdateCommandStep part of things outside knowing its the driver of this. With moving all property replacements, I realize this is a pretty big change. Is that something you or another member want to jump in to handle to make sure there are not any consequences I'm not noticing? Or do you want me to go ahead and attempt it and just report on my attempts?
I tried digging into this... and it requires a lot of knowledge about how everything fits together. I'm a little lost at how I could successfully move the replacement of the properties to AbstractUpdateCommandStep. I would definitely appreciate help here.
Also curious to hear feedback about having a more static deployment ID as I discussed above with the UUID/ numeric version of UUID that does not change with the resets!
Okay... I realized another facet of this last night. Continuing down this path, the replacement of the deployment ID is being done before a checksum is calculated for a changeset. This means that a changeset using this property will always appear as having changed since the changeset would be different because the deployment ID will be different on every execution. This is obviously not ideal (runOnce changesets with this property will cause errors on every run other than the first run and runOnChange changesets will run every single time).
So, it may actually be preferable to replace the deployment ID property (and maybe the other properties I am introducing as well?) later in the process (after the checksums have been calculated and after the lock has been acquired/final deployment ID is generated). Do you agree?
Is there a mechanism in place already to do property replacements later on in the update process?
@filipelautert Okay... sorry for all the pings!
But I think I have a viable solution. Instead of moving the replacement code later in the overall command process (I couldn't see a clean way to do this without A LOT of refactoring), I instead moved the lock acquistion code sooner in the overall command process.
I did this by altering the DbUrlConnectionCommandStep to acquire a changelog lock as soon as a connection to the database is established (and generate a deployment ID so it is available). You'll notice in the DbUrlConnectionCommandStep that I include this lock acquisition for 4 classes. This is because those are the four classes that call the waitForLock method. If there is a better/cleaner way of determining when to acquire the changelog lock and when not to at this step, I'm all for changing this method!
I ran the full test suite and had to update a couple existing tests because of the logic moving around, but overall this seems to be a pretty small change in the grand scheme of things.
Please take a look and let me know if this is viable or not!
I also haven't provided any new tests... want to make sure this process is accepted for trying to write tests (if there are any tests you want me to write...)
It may be cleaner to add a LockServiceCommandStep step to the update command's pipeline right after the DbUrlConnectionCommandStep step... but I can't figure out how to add a step to a pipeline like that! Happy to rework this to implement that if you or someone would be able to help me there!
And, after sleeping on it, I figured out the command pipeline is controlled by the dependencies listed by the command step class.
When I looked, the UpdateCommandStep class was overriding its parent classes' dependencies (AbstractUpdateCommandStep) and leaving out the LockService class. By switching the UpdateCommandStep to be based on the AbstractUpdateCommandStep classes' dependencies (and therefore including the LockService), this solved the problem other than needing to generate the deployment ID when the LockService.run method was executed.
This is solved the problem of the previous tests I had to change. I was able to change them back, so I am not affecting any commands that I shouldn't be affecting.
I think the only thing left to consider is the fact that anything using the LIQUIBASE_EXECUTION_DEPLOYMENT_ID property will have a different checksum every time it is run. Is this acceptable or not? If this is acceptable... I believe I am ready to take this PR out of DRAFT status and move forward with a full review!
@jasonlyle88 right now AbstractUpdateCommandStep uses the LockService as you mentioned, but we had to remove it from UpdateCommandStep because of the isFastCheckEnabled && isUpToDate logic. -> https://github.com/liquibase/liquibase/pull/4427 . I don't like the way that it is done too, but so far I can't think of a way to move fast check before the locking as it requires a lot of stuff generated by the other steps.
Oh, and reviewing that I'm starting to think that issue https://github.com/liquibase/liquibase/issues/5254 may be caused by the same reason.
Still thinking...
@filipelautert Ah, yup, I'm undoing a good amount of what was done in #4427 .
I'm unfamiliar with fast check, so I will leave that side in your capable hands.
Looking at #5254, I would agree that your hunch is probably correct!
Please let me know if I can be of assistance in any way here!
@filipelautert Any comments on using a UUID type value for the deployment ID?
This would allow me to generate the deployment ID independent of the change log service and it would not need to be reset. Without being reset and with the value being all but guaranteed to be unique, this value can be put somewhere else in the code that is more static (so resetting the changelog service doesn't reset the deployment ID).
Then all my current problems go away, because I can access the deployment ID without any worries of it changing on me.
Thoughts?
This would allow me to generate the deployment ID independent of the change log service and it would not need to be reset. Without being reset and with the value being all but guaranteed to be unique, this value can be put somewhere else in the code that is more static (so resetting the changelog service doesn't reset the deployment ID).
It would mean generating 1 deploymentId as first step right? I think I've a simpler idea that doesn't involve changing dbcl table. I implemented and merged a fix for the multithreading Scope object here -> https://github.com/liquibase/liquibase/pull/5617/files . So now every thread has it's own scope.
Maybe we could generate a single deploymentId and add it to root scope, and this way we will have only one per liquibase instance .
This would imply:
- changing Scope.getCurrentScope() to define the deploymentId
- changing AbstractChangeLogHistoryService.generateDeploymentId() to be a noop method and getDeploymentId() would return Scope value for deploymentId .
Would it resolve your problems?
@filipelautert Very cool. Thats pretty much what I was proposing, you just now have a good logical place for where this should occur! However, I think my question remains about convertin from a time based deployment ID to a UUID based deployment ID. For example, in this new root scope way of doing things, what happens if 2 different systems initialize liquibase within the same second?
Right now, the deployment ID (the final one, the one that gets used) gets calculated after a lock is obtained. This ensures that the deployment IDs are unique because only 1 person can have a lock at a given time.
In your model, wouldn't the deployment ID be calculated based on the start of liquibase, so if two instances are started within a second of eachother, they would end up with the same deployment ID? That was my reasoning for making it a UUID vs the current time based methodology.
Basically, you solved 1 part of the problem, and I still think the UUID solves the other part of the problem.
Thoughts?
Basically, you solved 1 part of the problem, and I still think the UUID solves the other part of the problem.
Thoughts?
I tried to understand why it was created date-based and seems there is no apparent reason for that - which is good as we have other columns to be used to sort it.
Fitting the uuid into the existing 10 characters space would be an easier solution. Changing the size to fit a complete uuid would involve changes in StandardChangeLogHistoryService.init and would slow down this PR (maybe it could be a second PR only to do this change?) . Anyway I like this idea as UUIDs are an universal (pun intended) thing.
Unfortunately, I don't think its possible to reliably compress/encode down the length of a UUID to 10 characters.
As such, I'm happy to proceed however you think it best. If you want me to create a separate PR to implement the DEPLOYMENT_ID field length change (and also the switch to UUID?), I can do that and put this on hold until that is accepted. Or I can include it in this PR.
Let me know what you and the team think best!
@jasonlyle88 so to unblock this PR how about getting rid of all of the deployment id references so we can move ahead with the other 3 substitutions , and then do all the deployment id improvements (scope, UUID, replacement, etc) in another PR?
I can do that. I'll get that done sometime soon and when I do I'll pull the PR out or DRAFT status and also work on the secondary PR after that.
Thanks for all your help here!
@filipelautert I have removed this from the draft status. I have manually tested the changes... however, I couldn't figure out how to write tests for this.
Since this has to happen at the time the nodes are parsing (because this is a property which gets replaced during parsing) I essentially need to run a whole changelog that has outputs of the new execution parameters and then have the test look at the output from the run and make sure it contains the correct data. But I couldn't find any examples of how this should be done and could not figure it out myself.
Any help here would be appreciated!
Since this has to happen at the time the nodes are parsing (because this is a property which gets replaced during parsing) I essentially need to run a whole changelog that has outputs of the new execution parameters and then have the test look at the output from the run and make sure it contains the correct data. But I couldn't find any examples of how this should be done and could not figure it out myself.
Any help here would be appreciated!
@jasonlyle88 I believe we would need a new integration test, something like what we have on DiffChangelogIntegrationTest or DiffIntegrationTest inside liquibase-integration-tests. In those tests we are able to run a CommandScope and evaluate the results from the execution so you could validate the 3 parameters used . What do you think? (and sorry for the delay to answer you)
Hey @filipelautert . Thanks for pointing me in the correct direction. I'd never actually looked at the integration tests before, and that makes sense. I've made a first attempt at my integration test, however I can't run the integration test locally and I don't believe I can trigger the integration tests on github.
I've got everything setup, I'm just not sure what the output variable will contain... and therefore unsure what I need to do for my assert.
Would you be able to help/guide me any more? Thanks!
@jasonlyle88 sorry, I missed your last comment. @MalloD12 could you provide guidance on how to add integration tests here?
git pull https://github.com/jasonlyle88/liquibase.git changeset-execution-information-properties
Hey @jasonlyle88,
If you want to run/debug your tests what you can do is first to have a liquibase.sdk.local.yaml file (this is the location of this file: liquibase-extension-testing/src/main/resources/liquibase.sdk.local.yaml and its content is similar to liquibase.sdk.yaml file). Basically the local version will override any property you define in the original version. For example my local version looks like:
liquibase:
sdk:
testSystem:
default:
username: lbuser
password: LiquibasePass1
catalog: lbcat
altCatalog: lbcat2
altSchema: lbschem2
altTablespace: liquibase2
keepRunning: true
test: postgresql
Besides this configuration, the only other pre-requisite to run this test in Postgres would be to have your instance already running. I have a docker container running for that.
Now, specifically talking about the test, I ran it and got a complain that the id column should be either a smallint, integer, or bigint. So, we might need to udpate the below column:
<column name="id" type="number" autoIncrement="true" />
From the then block:
CommandResults results = executeSql.execute()
String output = results.getResult("output") as String
System.out.println("JML TEST: '" + output + "'")
assert 1 == 1
I would move the first two lines to the when section to leave in this block the assertions we want to make for our test. I think then you can manipulate the output string to add the validation/s you want to perform.
Please let me know if this is not clear enough, and I'll do my best to clarify any question you have.
Thanks, Daniel.
Apologies, things got a little busy for a bit, and I'm finally able to get back into some liquibase work.
@MalloD12 , everything you said makes sense.
However... when I came back and started trying to do work, i started running into some issues.
I synced upstream main to my repo and was no longer able to build. I thought maybe I screwed something up, so I cloned a fresh repository and ran mvn clean install (following this guide) and it failed again. So, I thought something with my system got messed up. Since I knew I could run my branch before, I pulled a clean version of my branch, without the latest pull from upstream master, and did an mvn clean install on that... and it worked.
So old code is building just fine for me, but new code won't build, even with 0 modifications from master.
I've tried running mvn clean install on master with Temurin (adoptium) 11, 17, and 21. All fail.
The following is the end of the output from the execution
[INFO]
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] HsqlIntegrationTest>AbstractIntegrationTest.testClearChecksums:843 » CommandExecution liquibase.exception.LiquibaseException: liquibase.exception.MigrationFailedException: Migration failed for changeset changelogs/hsqldb/complete/root.changelog.xml::1.1::nvoxland:
Reason: liquibase.exception.DatabaseException: object name already exists: SEQ_PERSON in statement [CREATE SEQUENCE PUBLIC.seq_person AS BIGINT] [Failed SQL: (-5504) CREATE SEQUENCE PUBLIC.seq_person AS BIGINT]
[ERROR] HsqlIntegrationTest>AbstractIntegrationTest.testDbDoc:926 » CommandExecution liquibase.exception.LiquibaseException: liquibase.exception.MigrationFailedException: Migration failed for changeset changelogs/hsqldb/complete/root.changelog.xml::1.1::nvoxland:
Reason: liquibase.exception.DatabaseException: object name already exists: SEQ_PERSON in statement [CREATE SEQUENCE PUBLIC.seq_person AS BIGINT] [Failed SQL: (-5504) CREATE SEQUENCE PUBLIC.seq_person AS BIGINT]
[ERROR] HsqlIntegrationTest>AbstractIntegrationTest.testDiff:638->AbstractIntegrationTest.runCompleteChangeLog:356->AbstractIntegrationTest.runChangeLogFile:368 » CommandExecution liquibase.exception.LiquibaseException: liquibase.exception.MigrationFailedException: Migration failed for changeset changelogs/hsqldb/complete/root.changelog.xml::1.1::nvoxland:
Reason: liquibase.exception.DatabaseException: object name already exists: SEQ_PERSON in statement [CREATE SEQUENCE PUBLIC.seq_person AS BIGINT] [Failed SQL: (-5504) CREATE SEQUENCE PUBLIC.seq_person AS BIGINT]
[ERROR] HsqlIntegrationTest>AbstractIntegrationTest.testGenerateChangeLogWithNoChanges:1066->AbstractIntegrationTest.runCompleteChangeLog:356->AbstractIntegrationTest.runChangeLogFile:368 » CommandExecution liquibase.exception.LiquibaseException: liquibase.exception.MigrationFailedException: Migration failed for changeset changelogs/hsqldb/complete/root.changelog.xml::1.1::nvoxland:
Reason: liquibase.exception.DatabaseException: object name already exists: SEQ_PERSON in statement [CREATE SEQUENCE PUBLIC.seq_person AS BIGINT] [Failed SQL: (-5504) CREATE SEQUENCE PUBLIC.seq_person AS BIGINT]
[ERROR] HsqlIntegrationTest>AbstractIntegrationTest.testRerunDiffChangeLog:660->AbstractIntegrationTest.runCompleteChangeLog:356->AbstractIntegrationTest.runChangeLogFile:368 » CommandExecution liquibase.exception.LiquibaseException: liquibase.exception.MigrationFailedException: Migration failed for changeset changelogs/hsqldb/complete/root.changelog.xml::1.1::nvoxland:
Reason: liquibase.exception.DatabaseException: object name already exists: SEQ_PERSON in statement [CREATE SEQUENCE PUBLIC.seq_person AS BIGINT] [Failed SQL: (-5504) CREATE SEQUENCE PUBLIC.seq_person AS BIGINT]
[ERROR] HsqlIntegrationTest>AbstractIntegrationTest.testRollbackToChange:902 » CommandExecution liquibase.exception.LiquibaseException: liquibase.exception.MigrationFailedException: Migration failed for changeset changelogs/hsqldb/rollback/rollbackable.changelog.xml::8a::nvoxland:
Reason: liquibase.exception.DatabaseException: object name already exists: SEQ_PUBLICATION_ID in statement [CREATE SEQUENCE PUBLIC.seq_publication_id AS BIGINT] [Failed SQL: (-5504) CREATE SEQUENCE PUBLIC.seq_publication_id AS BIGINT]
[ERROR] HsqlIntegrationTest>AbstractIntegrationTest.testRowCountPrecondition:1145->AbstractIntegrationTest.runChangeLogFile:368 » CommandExecution liquibase.exception.LiquibaseException: liquibase.exception.MigrationFailedException: Migration failed for changeset changelogs/common/common.tests.changelog.xml::seq-test::nvoxland:
Reason: liquibase.exception.DatabaseException: object name already exists: SEQTESTONALL in statement [CREATE SEQUENCE PUBLIC.seqtestonall AS BIGINT] [Failed SQL: (-5504) CREATE SEQUENCE PUBLIC.seqtestonall AS BIGINT]
[ERROR] HsqlIntegrationTest>AbstractIntegrationTest.testRunUpdateOnOldChangelogTableFormat:423 » CommandExecution liquibase.exception.LiquibaseException: liquibase.exception.MigrationFailedException: Migration failed for changeset changelogs/hsqldb/complete/root.changelog.xml::1.1::nvoxland:
Reason: liquibase.exception.DatabaseException: object name already exists: SEQ_PERSON in statement [CREATE SEQUENCE PUBLIC.seq_person AS BIGINT] [Failed SQL: (-5504) CREATE SEQUENCE PUBLIC.seq_person AS BIGINT]
[ERROR] HsqlIntegrationTest>AbstractIntegrationTest.testSnapshotReportsAllObjectTypes:329->AbstractIntegrationTest.runCompleteChangeLog:356->AbstractIntegrationTest.runChangeLogFile:368 » CommandExecution liquibase.exception.LiquibaseException: liquibase.exception.MigrationFailedException: Migration failed for changeset changelogs/hsqldb/complete/root.changelog.xml::1.1::nvoxland:
Reason: liquibase.exception.DatabaseException: object name already exists: SEQ_PERSON in statement [CREATE SEQUENCE PUBLIC.seq_person AS BIGINT] [Failed SQL: (-5504) CREATE SEQUENCE PUBLIC.seq_person AS BIGINT]
[ERROR] HsqlIntegrationTest>AbstractIntegrationTest.testTableExistsPreconditionTableNameMatch:1111->AbstractIntegrationTest.runChangeLogFile:368 » CommandExecution liquibase.exception.LiquibaseException: liquibase.exception.MigrationFailedException: Migration failed for changeset changelogs/common/common.tests.changelog.xml::seq-test::nvoxland:
Reason: liquibase.exception.DatabaseException: object name already exists: SEQTESTONALL in statement [CREATE SEQUENCE PUBLIC.seqtestonall AS BIGINT] [Failed SQL: (-5504) CREATE SEQUENCE PUBLIC.seqtestonall AS BIGINT]
[ERROR] HsqlIntegrationTest>AbstractIntegrationTest.testTableIsEmptyPrecondition:1121->AbstractIntegrationTest.runChangeLogFile:368 » CommandExecution liquibase.exception.LiquibaseException: liquibase.exception.MigrationFailedException: Migration failed for changeset changelogs/common/common.tests.changelog.xml::seq-test::nvoxland:
Reason: liquibase.exception.DatabaseException: object name already exists: SEQTESTONALL in statement [CREATE SEQUENCE PUBLIC.seqtestonall AS BIGINT] [Failed SQL: (-5504) CREATE SEQUENCE PUBLIC.seqtestonall AS BIGINT]
[ERROR] HsqlIntegrationTest>AbstractIntegrationTest.testTag:628 » CommandExecution liquibase.exception.LiquibaseException: liquibase.exception.MigrationFailedException: Migration failed for changeset changelogs/hsqldb/complete/root.changelog.xml::1.1::nvoxland:
Reason: liquibase.exception.DatabaseException: object name already exists: SEQ_PERSON in statement [CREATE SEQUENCE PUBLIC.seq_person AS BIGINT] [Failed SQL: (-5504) CREATE SEQUENCE PUBLIC.seq_person AS BIGINT]
[ERROR] HsqlIntegrationTest>AbstractIntegrationTest.testTagEmptyDatabase:861 » CommandExecution liquibase.exception.LiquibaseException: liquibase.exception.MigrationFailedException: Migration failed for changeset changelogs/hsqldb/rollback/rollbackable.changelog.xml::8a::nvoxland:
Reason: liquibase.exception.DatabaseException: object name already exists: SEQ_PUBLICATION_ID in statement [CREATE SEQUENCE PUBLIC.seq_publication_id AS BIGINT] [Failed SQL: (-5504) CREATE SEQUENCE PUBLIC.seq_publication_id AS BIGINT]
[ERROR] HsqlIntegrationTest>AbstractIntegrationTest.testUpdateChangelogChecksum:1201 » CommandExecution liquibase.exception.LiquibaseException: liquibase.exception.MigrationFailedException: Migration failed for changeset changelogs/hsqldb/complete/included.changelog.xml::1.2::nvoxland:
Reason: liquibase.exception.DatabaseException: object name already exists: SEQ_NEWS in statement [CREATE SEQUENCE PUBLIC.seq_news AS BIGINT] [Failed SQL: (-5504) CREATE SEQUENCE PUBLIC.seq_news AS BIGINT]
[ERROR] HsqlIntegrationTest>AbstractIntegrationTest.testUpdateClearUpdate:557 » CommandExecution liquibase.exception.LiquibaseException: liquibase.exception.MigrationFailedException: Migration failed for changeset changelogs/hsqldb/complete/root.changelog.xml::1.1::nvoxland:
Reason: liquibase.exception.DatabaseException: object name already exists: SEQ_PERSON in statement [CREATE SEQUENCE PUBLIC.seq_person AS BIGINT] [Failed SQL: (-5504) CREATE SEQUENCE PUBLIC.seq_person AS BIGINT]
[ERROR] HsqlIntegrationTest>AbstractIntegrationTest.testUpdateTwice:543 » CommandExecution liquibase.exception.LiquibaseException: liquibase.exception.MigrationFailedException: Migration failed for changeset changelogs/hsqldb/complete/root.changelog.xml::1.1::nvoxland:
Reason: liquibase.exception.DatabaseException: object name already exists: SEQ_PERSON in statement [CREATE SEQUENCE PUBLIC.seq_person AS BIGINT] [Failed SQL: (-5504) CREATE SEQUENCE PUBLIC.seq_person AS BIGINT]
[INFO]
[ERROR] Tests run: 1137, Failures: 0, Errors: 16, Skipped: 602
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Liquibase Parent 0-SNAPSHOT:
[INFO]
[INFO] Liquibase Parent ................................... SUCCESS [ 42.811 s]
[INFO] liquibase-standard ................................. SUCCESS [02:00 min]
[INFO] liquibase-cli ...................................... SUCCESS [ 9.873 s]
[INFO] liquibase-snowflake ................................ SUCCESS [ 9.132 s]
[INFO] Liquibase .......................................... SUCCESS [ 1.644 s]
[INFO] liquibase-maven-plugin ............................. SUCCESS [ 21.132 s]
[INFO] liquibase-cdi-jakarta .............................. SUCCESS [ 11.239 s]
[INFO] liquibase-cdi ...................................... SUCCESS [ 9.956 s]
[INFO] liquibase-extension-testing ........................ SUCCESS [ 11.557 s]
[INFO] liquibase-integration-tests ........................ FAILURE [01:28 min]
[INFO] liquibase-dist ..................................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 05:26 min
[INFO] Finished at: 2024-05-02T17:58:29-04:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.2.5:test (default-test) on project liquibase-integration-tests:
[ERROR]
[ERROR] Please refer to /Users/jlyle/temp/liquibase/liquibase-integration-tests/target/surefire-reports for the individual test results.
[ERROR] Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn <args> -rf :liquibase-integration-tests
I have also attached the associated file from the surefire-reports directory
liquibase.dbtest.hsqldb.HsqlIntegrationTest.txt
@MalloD12 @filipelautert any thoughts on this? Can't really do any development if I can't build/run/test anything 😩
@MalloD12 @filipelautert Some further information:
I deleted the liquibase-integration-tests/src/test/java/liquibase/dbtest/hsqldb/HsqlIntegrationTest.java file that was failing and was able to run mvn clean install without issues. However, I am still not able to test as expected.
Following the setup guide, (specifically the "Running the CLI In Your IDE" section) used to also work fine for me. Now when I attempt to run it, it also fails. Below is the error output:
/Library/Java/JavaVirtualMachines/temurin-21.jdk/Contents/Home/bin/java -javaagent:/Applications/IntelliJ IDEA.app/Contents/lib/idea_rt.jar=53163:/Applications/IntelliJ IDEA.app/Contents/bin -Dfile.encoding=UTF-8 -Dsun.stdout.encoding=UTF-8 -Dsun.stderr.encoding=UTF-8 -classpath /Users/jlyle/Personal/projects/liquibase/liquibase/liquibase-cli/target/classes:/Users/jlyle/Personal/projects/liquibase/liquibase/liquibase-standard/target/classes:/Users/jlyle/.m2/repository/com/opencsv/opencsv/5.9/opencsv-5.9.jar:/Users/jlyle/.m2/repository/org/apache/commons/commons-lang3/3.13.0/commons-lang3-3.13.0.jar:/Users/jlyle/.m2/repository/org/apache/commons/commons-text/1.11.0/commons-text-1.11.0.jar:/Users/jlyle/.m2/repository/org/apache/commons/commons-collections4/4.4/commons-collections4-4.4.jar:/Users/jlyle/.m2/repository/org/yaml/snakeyaml/2.2/snakeyaml-2.2.jar:/Users/jlyle/.m2/repository/javax/xml/bind/jaxb-api/2.3.1/jaxb-api-2.3.1.jar:/Users/jlyle/.m2/repository/info/picocli/picocli/4.7.5/picocli-4.7.5.jar liquibase.integration.commandline.LiquibaseLauncher update
May 02, 2024 8:22:55 PM liquibase.servicelocator
INFO: Cannot load service
java.util.ServiceConfigurationError: liquibase.configuration.AutoloadedConfigurations: Provider com.datical.liquibase.ext.config.ReportConfiguration could not be instantiated
at java.base/java.util.ServiceLoader.fail(ServiceLoader.java:586)
at java.base/java.util.ServiceLoader$ProviderImpl.newInstance(ServiceLoader.java:813)
at java.base/java.util.ServiceLoader$ProviderImpl.get(ServiceLoader.java:729)
at java.base/java.util.ServiceLoader$3.next(ServiceLoader.java:1403)
at liquibase.servicelocator.StandardServiceLocator.findInstances(StandardServiceLocator.java:24)
at liquibase.configuration.LiquibaseConfiguration.init(LiquibaseConfiguration.java:61)
at liquibase.Scope.getCurrentScope(Scope.java:103)
at liquibase.command.CommandArgumentDefinition$Building.build(CommandArgumentDefinition.java:299)
at liquibase.command.core.CalculateChecksumCommandStep.<clinit>(CalculateChecksumCommandStep.java:52)
at liquibase.integration.commandline.LiquibaseCommandLine.<init>(LiquibaseCommandLine.java:109)
at liquibase.integration.commandline.LiquibaseCommandLine.main(LiquibaseCommandLine.java:94)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at liquibase.integration.commandline.LiquibaseLauncher.main(LiquibaseLauncher.java:116)
Caused by: java.lang.NoSuchMethodError: 'java.lang.Boolean liquibase.util.ValueHandlerUtil.booleanValueHandler(java.lang.Object)'
at com.datical.liquibase.ext.config.ReportConfiguration.<clinit>(ReportConfiguration.java:28)
at java.base/jdk.internal.misc.Unsafe.ensureClassInitialized0(Native Method)
at java.base/jdk.internal.misc.Unsafe.ensureClassInitialized(Unsafe.java:1160)
at java.base/jdk.internal.reflect.MethodHandleAccessorFactory.ensureClassInitialized(MethodHandleAccessorFactory.java:300)
at java.base/jdk.internal.reflect.MethodHandleAccessorFactory.newConstructorAccessor(MethodHandleAccessorFactory.java:103)
at java.base/jdk.internal.reflect.ReflectionFactory.newConstructorAccessor(ReflectionFactory.java:200)
at java.base/java.lang.reflect.Constructor.acquireConstructorAccessor(Constructor.java:549)
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486)
at java.base/java.util.ServiceLoader$ProviderImpl.newInstance(ServiceLoader.java:789)
... 12 more
May 02, 2024 8:22:55 PM liquibase.servicelocator
INFO: Cannot load service
java.util.ServiceConfigurationError: liquibase.configuration.AutoloadedConfigurations: Provider com.datical.liquibase.ext.config.DatabaseChangelogHistoryConfiguration could not be instantiated
at java.base/java.util.ServiceLoader.fail(ServiceLoader.java:586)
at java.base/java.util.ServiceLoader$ProviderImpl.newInstance(ServiceLoader.java:813)
at java.base/java.util.ServiceLoader$ProviderImpl.get(ServiceLoader.java:729)
at java.base/java.util.ServiceLoader$3.next(ServiceLoader.java:1403)
at liquibase.servicelocator.StandardServiceLocator.findInstances(StandardServiceLocator.java:24)
at liquibase.configuration.LiquibaseConfiguration.init(LiquibaseConfiguration.java:61)
at liquibase.Scope.getCurrentScope(Scope.java:103)
at liquibase.command.CommandArgumentDefinition$Building.build(CommandArgumentDefinition.java:299)
at liquibase.command.core.CalculateChecksumCommandStep.<clinit>(CalculateChecksumCommandStep.java:52)
at liquibase.integration.commandline.LiquibaseCommandLine.<init>(LiquibaseCommandLine.java:109)
at liquibase.integration.commandline.LiquibaseCommandLine.main(LiquibaseCommandLine.java:94)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at liquibase.integration.commandline.LiquibaseLauncher.main(LiquibaseLauncher.java:116)
Caused by: java.lang.NoSuchMethodError: 'java.lang.Boolean liquibase.util.ValueHandlerUtil.booleanValueHandler(java.lang.Object)'
at com.datical.liquibase.ext.config.DatabaseChangelogHistoryConfiguration.<clinit>(DatabaseChangelogHistoryConfiguration.java:56)
at java.base/jdk.internal.misc.Unsafe.ensureClassInitialized0(Native Method)
at java.base/jdk.internal.misc.Unsafe.ensureClassInitialized(Unsafe.java:1160)
at java.base/jdk.internal.reflect.MethodHandleAccessorFactory.ensureClassInitialized(MethodHandleAccessorFactory.java:300)
at java.base/jdk.internal.reflect.MethodHandleAccessorFactory.newConstructorAccessor(MethodHandleAccessorFactory.java:103)
at java.base/jdk.internal.reflect.ReflectionFactory.newConstructorAccessor(ReflectionFactory.java:200)
at java.base/java.lang.reflect.Constructor.acquireConstructorAccessor(Constructor.java:549)
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486)
at java.base/java.util.ServiceLoader$ProviderImpl.newInstance(ServiceLoader.java:789)
... 12 more
May 02, 2024 8:22:56 PM liquibase.servicelocator
INFO: Cannot load service
java.util.ServiceConfigurationError: liquibase.command.CommandStep: Provider com.datical.liquibase.ext.command.helpers.ChecksAutoEnableNewChecksArgument could not be instantiated
at java.base/java.util.ServiceLoader.fail(ServiceLoader.java:586)
at java.base/java.util.ServiceLoader$ProviderImpl.newInstance(ServiceLoader.java:813)
at java.base/java.util.ServiceLoader$ProviderImpl.get(ServiceLoader.java:729)
at java.base/java.util.ServiceLoader$3.next(ServiceLoader.java:1403)
at liquibase.servicelocator.StandardServiceLocator.findInstances(StandardServiceLocator.java:24)
at liquibase.command.CommandFactory.findAllInstances(CommandFactory.java:247)
at liquibase.command.CommandFactory.getCommands(CommandFactory.java:163)
at liquibase.integration.commandline.LiquibaseCommandLine.getCommands(LiquibaseCommandLine.java:1112)
at liquibase.integration.commandline.LiquibaseCommandLine.buildPicoCommandLine(LiquibaseCommandLine.java:210)
at liquibase.integration.commandline.LiquibaseCommandLine.<init>(LiquibaseCommandLine.java:186)
at liquibase.integration.commandline.LiquibaseCommandLine.main(LiquibaseCommandLine.java:94)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at liquibase.integration.commandline.LiquibaseLauncher.main(LiquibaseLauncher.java:116)
Caused by: java.lang.NoSuchMethodError: 'java.lang.Boolean liquibase.util.ValueHandlerUtil.booleanValueHandler(java.lang.Object)'
at com.datical.liquibase.ext.command.helpers.ChecksAutoEnableNewChecksArgument.<clinit>(ChecksAutoEnableNewChecksArgument.java:23)
at java.base/jdk.internal.misc.Unsafe.ensureClassInitialized0(Native Method)
at java.base/jdk.internal.misc.Unsafe.ensureClassInitialized(Unsafe.java:1160)
at java.base/jdk.internal.reflect.MethodHandleAccessorFactory.ensureClassInitialized(MethodHandleAccessorFactory.java:300)
at java.base/jdk.internal.reflect.MethodHandleAccessorFactory.newConstructorAccessor(MethodHandleAccessorFactory.java:103)
at java.base/jdk.internal.reflect.ReflectionFactory.newConstructorAccessor(ReflectionFactory.java:200)
at java.base/java.lang.reflect.Constructor.acquireConstructorAccessor(Constructor.java:549)
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486)
at java.base/java.util.ServiceLoader$ProviderImpl.newInstance(ServiceLoader.java:789)
... 12 more
May 02, 2024 8:22:56 PM liquibase.servicelocator
INFO: Cannot load service
java.util.ServiceConfigurationError: liquibase.command.CommandStep: Provider com.datical.liquibase.ext.command.checks.ChecksRunCommandStep could not be instantiated
at java.base/java.util.ServiceLoader.fail(ServiceLoader.java:586)
at java.base/java.util.ServiceLoader$ProviderImpl.newInstance(ServiceLoader.java:813)
at java.base/java.util.ServiceLoader$ProviderImpl.get(ServiceLoader.java:729)
at java.base/java.util.ServiceLoader$3.next(ServiceLoader.java:1403)
at liquibase.servicelocator.StandardServiceLocator.findInstances(StandardServiceLocator.java:24)
at liquibase.command.CommandFactory.findAllInstances(CommandFactory.java:247)
at liquibase.command.CommandFactory.getCommands(CommandFactory.java:163)
at liquibase.integration.commandline.LiquibaseCommandLine.getCommands(LiquibaseCommandLine.java:1112)
at liquibase.integration.commandline.LiquibaseCommandLine.buildPicoCommandLine(LiquibaseCommandLine.java:210)
at liquibase.integration.commandline.LiquibaseCommandLine.<init>(LiquibaseCommandLine.java:186)
at liquibase.integration.commandline.LiquibaseCommandLine.main(LiquibaseCommandLine.java:94)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at liquibase.integration.commandline.LiquibaseLauncher.main(LiquibaseLauncher.java:116)
Caused by: java.lang.NoSuchMethodError: 'java.lang.Boolean liquibase.util.ValueHandlerUtil.booleanValueHandler(java.lang.Object)'
at com.datical.liquibase.ext.command.checks.ChecksRunCommandStep.<clinit>(ChecksRunCommandStep.java:211)
at java.base/jdk.internal.misc.Unsafe.ensureClassInitialized0(Native Method)
at java.base/jdk.internal.misc.Unsafe.ensureClassInitialized(Unsafe.java:1160)
at java.base/jdk.internal.reflect.MethodHandleAccessorFactory.ensureClassInitialized(MethodHandleAccessorFactory.java:300)
at java.base/jdk.internal.reflect.MethodHandleAccessorFactory.newConstructorAccessor(MethodHandleAccessorFactory.java:103)
at java.base/jdk.internal.reflect.ReflectionFactory.newConstructorAccessor(ReflectionFactory.java:200)
at java.base/java.lang.reflect.Constructor.acquireConstructorAccessor(Constructor.java:549)
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486)
at java.base/java.util.ServiceLoader$ProviderImpl.newInstance(ServiceLoader.java:789)
... 12 more
Exception in thread "main" java.lang.reflect.InvocationTargetException
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:118)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at liquibase.integration.commandline.LiquibaseLauncher.main(LiquibaseLauncher.java:116)
Caused by: java.lang.IllegalStateException: Unable to find CommandStep provider for class com.datical.liquibase.ext.command.helpers.ChecksAutoEnableNewChecksArgument
at liquibase.command.CommandFactory.lambda$whoProvidesClass$4(CommandFactory.java:121)
at java.base/java.util.Optional.orElseThrow(Optional.java:403)
at liquibase.command.CommandFactory.whoProvidesClass(CommandFactory.java:121)
at liquibase.command.CommandFactory.findDependenciesForCommand(CommandFactory.java:104)
at liquibase.command.CommandFactory.computePipelineForCommandDefinition(CommandFactory.java:77)
at liquibase.command.CommandFactory.getCommandDefinition(CommandFactory.java:50)
at liquibase.command.CommandFactory.getCommands(CommandFactory.java:175)
at liquibase.integration.commandline.LiquibaseCommandLine.getCommands(LiquibaseCommandLine.java:1112)
at liquibase.integration.commandline.LiquibaseCommandLine.buildPicoCommandLine(LiquibaseCommandLine.java:210)
at liquibase.integration.commandline.LiquibaseCommandLine.<init>(LiquibaseCommandLine.java:186)
at liquibase.integration.commandline.LiquibaseCommandLine.main(LiquibaseCommandLine.java:94)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
... 2 more
Process finished with exit code 1
So, I can't do any local testing. I'd love to get back to helping out, so hopefully someone knows what is going on!
Looks like I am running into 2 seperate issues with the mvn clean install failure and the failure to be able to run the CLI through my IDE. I checked out 1 commit on master at a time to determine where things stopped working and each problem can be attributed to a different PR. I have commented on those PRs in relation to the specific problem.
The mvn clean install issue is related to PR #5619
The inability to run the CLI through an IDE issue is related to PR #5729
Looks like I am running into 2 seperate issues with the
mvn clean installfailure and the failure to be able to run the CLI through my IDE. I checked out 1 commit on master at a time to determine where things stopped working and each problem can be attributed to a different PR. I have commented on those PRs in relation to the specific problem.The
mvn clean installissue is related to PR #5619 The inability to run the CLI through an IDE issue is related to PR #5729
@jasonlyle88 The valuehandler util is a good lead. Seems you version of liquibase-commercial is outdated - this one is downloaded from github package manager. Could you try to clear it from your m2 cache and build again?