Build state from Travis is misleading/incorrect

[BuzMack]

Most recent result scraped from https://travis-ci.com/github/gitlab4j/gitlab4j-api (which is apparently passing and is good) shows this at the end of the report:

[INFO] 
[INFO] Results:
[INFO] 
[ERROR] Failures: 
[ERROR]   TestSystemHooksApi.testAddSystemHook:67
[ERROR]   TestUserApi.testGetMemberships:463->assertMembershipEquals:479 expected:<project> but was:<null>
[ERROR] Errors: 
[ERROR]   TestGitLabLogin.testOauth2LoginWithCharArrayPassword:92 » GitLabApi <!DOCTYPE ...
[INFO] 
[ERROR] Tests run: 271, Failures: 2, Errors: 1, Skipped: 4
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  03:47 min
[INFO] Finished at: 2021-06-28T16:22:00Z
[INFO] ------------------------------------------------------------------------
The command "./mvnw integration-test -B -V -Dmaven.javadoc.skip=true" exited with 0.
after_script
12.00s$ ./mvnw docker:stop -B -Dgitlab.skip-docker-start=false
Done. Your build exited with 0.

271 tests ran 2 failures 1 error 4 skipped

If there are errors in the tests, how can this be considered a passing build? Yes it built, but integration tests which are supposed to validate the correctness of the build failed... so... this build is good?

Either the tests are broken, or the build is broken. In either case, there's a problem here.

[jabby]

@BuzMack Thanks for the report. I will investigate.

[BuzMack]

After doing a few tests across different GitLab versions, turns out there's an issue with GitLab itself. One of the listed failures in the above tests are caused by issue System hooks API does not honor "repository_update_events" attribute

[BuzMack]

Just noticed the updated builds... thanks for making changes to the tests, however there's still issues...

From TestSystemHooksApi class lines 72-87:

        hook.withPushEvents(false)
            .withTagPushEvents(true)
            .withMergeRequestsEvents(true)
            .withRepositoryUpdateEvents(false)
            .withEnableSslVerification(false);

        SystemHook updatedHook = gitLabApi.getSystemHooksApi().addSystemHook(TEST_HOOK_URL, TEST_SECRET_TOKEN, hook);
        assertNotNull(updatedHook);
        assertEquals(TEST_HOOK_URL, updatedHook.getUrl());
        assertFalse(hook.getPushEvents());
        assertTrue(hook.getTagPushEvents());
        assertTrue(hook.getMergeRequestsEvents());
        assertFalse(hook.getRepositoryUpdateEvents());
        assertFalse(hook.getEnableSslVerification());

        gitLabApi.getSystemHooksApi().deleteSystemHook(updatedHook);

The tailing asserts are asserting on hook and not on updatedHook, so are not testing the resulting object from the addSystemHook(...) call.

Also, while the tests in this case now pass, the underlying error is that GitLab's documentation and code are also incorrect... their implementation doesn't honor the given attribute for "repository_update_events"... I'd suggest commenting about that issue within the test code for future updates/corrections...

[jabby]

Can you provide a PR with the corresponding modification?

[BuzMack]

Sure, will do shortly.

[BuzMack]

PR 727 created (after having proxy issues with my work PC, so apologies for the delay).