ZOOKEEPER-3868: Completion leak in zookeeper_close

[tudor]

See https://issues.apache.org/jira/browse/ZOOKEEPER-3868

[tudor]

The s390x build failed with "An error occurred while generating the build script." which I'm pretty sure is unrelated to my change.

[tudor]

@eolivelli @ztzg @symat ping? :)

[eolivelli]

Did you consider adding a test case?

[symat]

@tudor there is a failed unit test reported by the maven job: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build-maven/2140/console

please take a look if you can, it seems to be related (Zookeeper_watchers::testDefaultSessionWatcher1).

I also copy the output in case the jenkins logs would be cleaned later:

....
     [exec] Zookeeper_multi::testWatch : elapsed 2010 : OK
     [exec] Zookeeper_multi::testSequentialNodeCreateInAsyncMulti : elapsed 2003 : OK
     [exec] Zookeeper_multi::testBigAsyncMulti : elapsed 3005 : OK
     [exec] Zookeeper_watchers::testDefaultSessionWatcher1FAIL: zktest-mt
     [exec] ==========================================
     [exec] 1 of 2 tests failed
     [exec] Please report to [email protected]
     [exec] ==========================================
     [exec] Makefile:1850: recipe for target 'check-TESTS' failed
     [exec] make[1]: Leaving directory '/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build-maven/zookeeper-client/zookeeper-client-c/target/c'
     [exec] Makefile:2106: recipe for target 'check-am' failed
     [exec] zktest-mt: /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build-maven/zookeeper-client/zookeeper-client-c/tests/ZKMocks.cc:288: SyncedBoolCondition DeliverWatchersWrapper::isDelivered() const: Assertion `i<1000' failed.
     [exec] /bin/bash: line 5:  8399 Aborted                 ZKROOT=/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build-maven/zookeeper-client/zookeeper-client-c/../.. CLASSPATH=$CLASSPATH:$CLOVER_HOME/lib/clover*.jar ${dir}$tst
     [exec] make[1]: *** [check-TESTS] Error 1
     [exec] make: *** [check-am] Error 2
....

[symat]

The s390x build failed with "An error occurred while generating the build script." which I'm pretty sure is unrelated to my change.

Yes, that travis job failure is strange... I haven't seen it yet. but if it happens again, we may need to debug it :)

(the unit test failure was reported by the jenkinsMaven job, and not by the travis job)

[symat]

maybe the unit test failure is unrelated... I try to re-trigger the jenkins job.

[symat]

retest maven build

[symat]

I see the same failure after the new unit test execution: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build-maven/2147/console

ZKMocks.cc:288: SyncedBoolCondition DeliverWatchersWrapper::isDelivered() const: Assertion 'i<1000' failed. for Zookeeper_watchers::testDefaultSessionWatcher1

[tudor]

That's weird, I'll investigate.

[ztzg]

Hi @tudor, @symat,

Sorry for the delay; I've been a bit "disconnected" and almost missed this. I'll have a look ASAP.

Cheers, -D

[ztzg]

Hi @tudor,

I’ve had a look at this; it's a nasty thing.

DeliverWatchersWrapper in tests/ZKMocks.cc tries to stop (some) threads by forcefully setting zh->close_requested=1 via terminateZookeeperThreads. adaptor_finish is not called when it does that, which means that terminate_completion remains "false." do_completion consequently does not exit as it expects.

(I haven't investigated further.)

HTH, -D