Make Quarkus*IntegrationTest runs use a dedicated "integration" profile for runtime config

[maxandersen]

Description

Currently QuarkusIntegrationTest, QuarkusMainIntegrationTest and NativeImageTest is defaulting to run with "prod" profile during their test runs.

This makes it tricky to use devservice features during integration testing as prod profile might intentionally or unintentionally set values that is not minded for testing.

Implementation ideas

Suggestion to solve that is to introduce a integration profile (named something like "inttest" or "it") that will be used to resolve runtime properties when running against prod built artifacts (which has prod profile in static init).

This allows running integration tests with use of devservices while still use prod profile for truly prod specific settings.

[yrodiere]

That's a good idea.

IMO we should also consider having test as the parent profile of the integration profile, so that test properties are automatically inherited by the integration profile`. That would address some concerns mentioned in #26279, mainly the confusion of users setting test properties and not seeing them applied in... tests.

[geoand]

Sounds reasonable indeed

[chberger]

I also like the idea very much. Anyway, I doubt that just introducing a separate "it" profile would help much. @geoand I appreciate the decision that integration tests use prod built artifacts, but it makes things really complicated :-)

IMHO with a new runtime profile u can just influence runtime configurations. This can already be achieved by using the system property -Dquarkus.test.native-image-profile=ti. It's a little bit confusing that this property changes the runtime profile of the artifacts under test, even in case when just testing jar artifacts or hotspot images.

Anyway, I have made the experience that using DevServices during ITs requires changes concerning build time configurations. For instance: I've tried to get a Postgres instance with some test data up and running. While Testcontainers works like a charm, I was not able to load test data with on-board functionality. Hibernates loading mechanism is configured by a build time property quarkus.hibernate-orm.sql-load-script and the same is true for Flyway quarkus.flyway.locations.

It's almost impossible to use DevServices during integration tests (@QuarkusIntegrationTest), due to the fact that prod build artifacts are being used. I believe this is even a design issue because on the one hand, you like to have a prod built on the other hand you like to have test infrastructure, both controlled with a single profile configuration ...

Long story short: As a Quarkus user the following points are very confusing:

• Integration tests are being executed with build artifacts compiled and running in the prod profile (Although I understand that it makes totally sense).
• DevServices are active during integration tests by default, although the artifacts under test run with the prod profile. I would have expected that DevServices are only enabled in dev & test profile.
• The system property -Dquarkus.test.native-image-profile=ti works for non native images as well
• The fact that it's really tricky to use DevServices during ITs (IMHO: DevServices is maybe also not the best name when you want to use it to ramp up infrastructure services during ITs)

Overall my testing experience with ITs is really frustrating. Although I really appreciated your hard work and I love to work with Quarkus ..

[geoand]

Thanks for your valuable feedback. I'll think this through some more and see what we can do

[yrodiere]

Hibernates loading mechanism is configured by a build time property quarkus.hibernate-orm.sql-load-script

See also https://github.com/quarkusio/quarkus/issues/21866#issuecomment-1046997899

[ayhanap]

I am also having difficulty with build-time config like quarkus.liquibase-mongodb.change-log while doing @QuarkusIntegrationTest. Even if I use quarkus.test.profile=test or quarkus.test.native-image-profile=test, because quarkus.liquibase-mongodb.change-log is a build time config, setting runtime quarkus profile does not help.

What I am trying to do is just to populate dev-services DB for integration-tests. But since I want to populate it only on integration-tests or tests in general, I cannot use same changelog in prod config.

As a workaround I just used same changelog file for both test and prod but with parameters. Because the change-log-parameters is a runtime config, I can now specify which change-logs to run for test DB.

quarkus.liquibase-mongodb.change-log=liquibase/changelog.xml without any profile %test.quarkus.liquibase-mongodb.change-log-parameters.integration-test=true

--changelog.xml

<changeSet id="1" author="alex">
     <preConditions onFail="WARN">
         <changeLogPropertyDefined property="integration-test"/>
     </preConditions>
     <ext:insertMany collectionName="user">
         <ext:documents>
             [
               {
                   "password": "xxx",
                   "email": "xxx",
                   "role": "org-admin"
               }
             ]
         </ext:documents>
     </ext:insertMany>
 </changeSet>

[geoand]

Thanks for the input

[knutwannheden]

Interesting. This is also matches what we are doing in our Quarkus projects. Instead of using @QuarkusIntegrationTest we built our own JUnit5 extension to run tests against the application running as Docker container. We are however looking into migrating to @QuarkusIntegrationTest. For that we would however certainly want to use a dedicated config profile.

[jacksoncastro]

Hello everyone!

I basically had the same problem. When running ./mvnw verify Quarkus uses the profile "prod" instead of "test". This ends up forcing me to use test settings in the default profile.

Can anyone help?

[geoand]

For now, you can use the legacy quarkus.test.native-image-profile configuration property set whatever property you like

[jacksoncastro]

For now, you can use the legacy quarkus.test.native-image-profile configuration property set whatever property you like

Thanks for the answer.

I did the test and actually when running the final jar it added the -Dquarkus.profile=test argument, but the scope of the application remains prod.

Detailed code

/java/openjdk-17.0.2/bin/java -javaagent:$HOME/.m2/repository/org/jacoco/org.jacoco.agent/0.8.8/org.jacoco.agent-0.8.8-runtime.jar=destfile=$HOME/quarkus-api/target/jacoco-quarkus.exec,append=true -Dquarkus.http.port=8081 -Dquarkus.http.ssl-port=8444 -Dtest.url=http://localhost:8081 -Dquarkus.log.file.path=$HOME/quarkus-api/target/quarkus.log -Dquarkus.log.file.enable=true -Dquarkus.profile=test -jar $HOME/quarkus-api/target/quarkus-app/quarkus-run.jar

What ends up generating this error:

Driver does not support the provided URL - which in turn is an H2 driver for test scope only.

Is it possible to run the integration tests with the test scope?

I ran it like this, but without success:

./mvnw clean verify -Dquarkus.test.native-image-profile=test

[agreedSkiing]

We are currently using a workaround for this to load our test data into our database through flyway with the help of @QuarkusTestResource on every @QuarkusIntegrationTest.

This test resource only does this and nothing more.

    @Override
    public Map<String, String> start() {
        return Map.of("quarkus.profile", "integration", "user.timezone", "UTC");
    }

And our application.properties

%dev.domain.common.flyway.locations=db/migration,db/test/data
%test.domain.common.flyway.locations=${%dev.domain.common.flyway.locations}
%integration.domain.common.flyway.locations=${%dev.domain.common.flyway.locations}

Sadly this does not work with all configuration like quarkus.hibernate-orm.jdbc.timezone.

A right almost forgot this project uses Hibernate reactive so we are using the workaround posted here https://github.com/quarkusio/quarkus/issues/10716#issuecomment-791470636

[agseijas]

Our problem is similar to this, we need some @QuarkusIntegrationTest to launch with some specific build-time properties to enable and configure our custom-made devservices (a database of sorts).

The (incomplete) workaround we had to do involves setting the needed properties in maven-failsafe systemPropertyVariables.

This makes it build successfully but makes running the test in the IDEs fail (as they don't use the maven-failsafe config), so in order to run them from the IDE you need to add the properties on each launch configuration. Painful, but it works.

The proposal here goes in the right direction and will solve the issue for us while we have no need for different configurations between tests (if the proposal is just 1 namespace like integrationtest). On the other hand I fear it might be not be enough unless we change the test setup approach and by ditching the devservices approach; as we might have many tests with this annotation and other similar (like native tests) which all need separate + build-time configurations (for example to seed some data).

If I would make a request to resolve our situation (and there is a high chance that we overlook something and the approach is plainly wrong) it would be to be able to tell to the build process the build-time properties per integration-test class (and currently the build just launches once, unfortunately :cry: ). Example just for clarification:

@QuarkusIntegrationTest(buildTimeProperties = <Map.of(String, String)>)

Or a more simple using the approach in the opener @QuarkusIntegrationTest(profile = <String>

[geoand]

I should clarify that anything we do for this ticket, will be regarding runtime properties - build time properties aren't going to be altered, as that would fundamentally change what @QuarkusIntegrationTest is meant to do.

[chberger]

@geoand Any update on this?

Let me summarize my point once again:

I'm still struggling using DevServices in combination with QuarkusIntegrationTests. Ideally, as a Quarkus user I would expect that regardless of using QuarkusTests or QuarkusIntegrationTests I can use DevServices and their capabilities in the very same way.

However, this is not my experience, especially when it comes to QuarkusIntegrationTests. With these kind of tests I have the dilemma that the artifact under test has been built with the prod profile and thus it does not recognize any configurations assigned to the test profile. But I would love to have configurations like %test.quarkus.hibernate-orm.sql-load-script being recognized during IT test execution. If this would be the case, I could load test data in the very same way as I would do with QuarkusTests.

Additionally, I understand that I can change the runtime profile of the executable during QuarkusIntegrationTests, but this is useless as long as configurations are fixed at build time. And especially the configs I've checked in order to load test data are all fixed at build time:

• quarkus.hibernate-orm.sql-load-script
• quarkus.flyway.locations

So what's your opinion on how to handle this situation with the current Quarkus design?

Option 1: Don't use test configurations to load test data during ITs at all. Which results in a completely different test data loading strategy, at least in comparisons to QuarkusTests.

Option 2: Compile a dedicated IT executable with the test profile and re-compile it with prod once all tests have been passed. Personally, I don't like that approach at alI, because I don't test the real executable and I have to compile it twice.

Option 3: Relevant configurations must be switch to be runtime configs, so that we have a chance to recognize them when running the IT executable with a different runtime profile (quarkus.test.native-image-profile).

Would love to get some advice to get my ITs up and running soon.

Thanks in advance!

[geoand]

@geoand Any update on this?

I have not had time to work on this - hopefully I will for Quarkus 3.3.

Let me summarize my point once again:

Thanks for sharing your insights!

[chberger]

friendly reminder

[geoand]

I've doing some specific Quarkus related work that requires my full attention, so this has been moved further back in the queue until further notice

On Fri, Nov 3, 2023, 13:18 Christian Berger @.***> wrote:

friendly reminder

— Reply to this email directly, view it on GitHub https://github.com/quarkusio/quarkus/issues/24581#issuecomment-1792263296, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABBMDP5OD7ASSRC6JWH6633YCTHJBAVCNFSM5R25BIB2U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCNZZGIZDMMZSHE3A . You are receiving this because you were mentioned.Message ID: @.***>