Issue 5544 - Tessera as internal process

[gtebrean]

PR description

The key changes are in the TesseraInternalProcessTestHarness class. This starts Tessera nodes as processes.
For unzziping Tessera I created a gradle task.

Also got inspired from #4466

Fixed Issue(s)

Fixes #5544

[github-actions[bot]]

• [ ] I thought about documentation and added the doc-change-required label to this PR if updates are required.
• [x] I thought about the changelog and included a changelog update if required.
• [ ] If my PR includes database changes (e.g. KeyValueSegmentIdentifier) I have thought about compatibility and performed forwards and backwards compatibility tests

[non-fungible-nelson]

Hey @gtebrean - this would be a huge help in our CI simplification and testing processes. Anything you need to get this across the line?

[gtebrean]

Hey @gtebrean - this would be a huge help in our CI simplification and testing processes. Anything you need to get this across the line?

Hey Mat, the only thing left here is getting Tessera JVM started properly. Last time, I encountered some issues with the ports. I've been busy with work over the past few months, but now I'm getting back to this.

[macfarla]

had a quick look @gtebrean - are you able to run the privacy ATs locally? I tried with gradlew acceptanceTestPrivacy and got 122 failures out of 191 tests. (I could be missing something...)

Also added back in .circleci/config.yml the privacy AT task on PRs (we temporarily moved it out to the nightly build) so will see how that goes

[macfarla]

had a quick look @gtebrean - are you able to run the privacy ATs locally? I tried with gradlew acceptanceTestPrivacy and got 122 failures out of 191 tests. (I could be missing something...)

Also added back in .circleci/config.yml the privacy AT task on PRs (we temporarily moved it out to the nightly build) so will see how that goes

The errors I get locally are like this

PrivacyClusterAcceptanceTest > [TESSERA enclave type with EC encryptor] > org.hyperledger.besu.tests.acceptance.privacy.PrivacyClusterAcceptanceTest.canInteractWithMultiplePrivacyGroups[TESSERA enclave type with EC encryptor] FAILED
    java.lang.RuntimeException: java.lang.RuntimeException: java.util.concurrent.TimeoutException: Tessera did not start
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
        at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480)
        at java.base/java.util.concurrent.ForkJoinTask.getThrowableException(ForkJoinTask.java:562)
        at java.base/java.util.concurrent.ForkJoinTask.reportException(ForkJoinTask.java:591)
        at java.base/java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:689)
        at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159)
        at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173)
        at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
        at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)
        at org.hyperledger.besu.tests.acceptance.dsl.privacy.PrivacyCluster.startNodes(PrivacyCluster.java:71)
        at org.hyperledger.besu.tests.acceptance.dsl.privacy.PrivacyCluster.start(PrivacyCluster.java:52)
        at org.hyperledger.besu.tests.acceptance.dsl.privacy.PrivacyCluster.start(PrivacyCluster.java:48)
        at org.hyperledger.besu.tests.acceptance.privacy.PrivacyClusterAcceptanceTest.<init>(PrivacyClusterAcceptanceTest.java:104)

        Caused by:
        java.lang.RuntimeException: java.util.concurrent.TimeoutException: Tessera did not start
            at org.hyperledger.enclave.testutil.TesseraInternalProcessTestHarness.start(TesseraInternalProcessTestHarness.java:104)

[gtebrean]

@macfarla Thanks a lot for looking at! Yes I ran that locally and it worked, and also it passed trough the previous github pipelines. I didn't do anything special, I have the basic configurations. What I noticed today is that 2 from 3 builds are failing with 1, but no more that 2 tests, randomly. So this might be because new Tessera processes are unstable and yesterday I just got lucky with the builds.

Also the same behavoiur is for ./gradlew --no-daemon acceptanceTestPermissioning, it passed 7 out of 8 times. I'm investigating this further.

[macfarla]

@macfarla Thanks a lot for looking at! Yes I ran that locally and it worked, and also it passed trough the previous github pipelines. I didn't do anything special, I have the basic configurations. What I noticed today is that 2 from 3 builds are failing with 1, but no more that 2 tests, randomly. So this might be because new Tessera processes are unstable and yesterday I just got lucky with the builds.

Also the same behavoiur is for ./gradlew --no-daemon acceptanceTestPermissioning, it passed 7 out of 8 times. I'm investigating this further.

permissioning AT flakiness should be fixed by this https://github.com/hyperledger/besu/pull/6384

[macfarla]

only 1 out of 191 failed in CI so I guess there is some flakiness.

but I must be missing something locally too (maybe an assumption that is true in your local environment) -

➜  b2 git:(tessera_as_internal_process) ✗ gw unzipTessera
> Task :acceptance-tests:unzipTessera FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':acceptance-tests:unzipTessera'.
> Could not read '/Users/sm/workspace/b2/acceptance-tests/build/tessera/dist/tessera-dist-23.4.0.tar' as it does not exist.

[gtebrean]

I ran test commands multiples time, but I never got that issue. However when I tried to run only :acceptance-tests:unzipTessera it failed also for me. With last commit this should be fixed, but still remains the random process error that I'm still looking at.

[macfarla]

I get a slightly different error now running gw acceptanceTasks locally

DeployPrivateSmartContractAcceptanceTest > [RESTRICTED tx with TESSERA enclave and EC encryptor type] > org.hyperledger.besu.tests.acceptance.privacy.DeployPrivateSmartContractAcceptanceTest.deployingMustGiveValidReceiptAndCode[RESTRICTED tx with TESSERA enclave and EC encryptor type] FAILED
    java.lang.RuntimeException: org.web3j.protocol.exceptions.TransactionException: JsonRpcError thrown with code -50100. Message: Error communicating with enclave
        at org.hyperledger.besu.tests.acceptance.dsl.privacy.contract.DeployPrivateSmartContractTransaction.execute(DeployPrivateSmartContractTransaction.java:79)
        at org.hyperledger.besu.tests.acceptance.dsl.privacy.contract.DeployPrivateSmartContractTransaction.execute(DeployPrivateSmartContractTransaction.java:35)
        at org.hyperledger.besu.tests.acceptance.dsl.node.BesuNode.execute(BesuNode.java:810)
        at org.hyperledger.besu.tests.acceptance.dsl.privacy.PrivacyNode.execute(PrivacyNode.java:247)
        at org.hyperledger.besu.tests.acceptance.privacy.DeployPrivateSmartContractAcceptanceTest.deployingMustGiveValidReceiptAndCode(DeployPrivateSmartContractAcceptanceTest.java:62)

        Caused by:
        org.web3j.protocol.exceptions.TransactionException: JsonRpcError thrown with code -50100. Message: Error communicating with enclave
            at app//org.web3j.tx.Contract.executeTransaction(Contract.java:409)
            at app//org.web3j.tx.Contract.create(Contract.java:464)
            at app//org.web3j.tx.Contract.deploy(Contract.java:528)
            at app//org.web3j.tx.Contract.lambda$deployRemoteCall$10(Contract.java:730)
            at app//org.web3j.protocol.core.RemoteCall.send(RemoteCall.java:42)
            at app//org.hyperledger.besu.tests.acceptance.dsl.privacy.contract.DeployPrivateSmartContractTransaction.execute(DeployPrivateSmartContractTransaction.java:77)
            ... 4 more

[macfarla]

actually I do still get the "Tessera did not start" for later failures eg

PrivacyClusterAcceptanceTest > [TESSERA enclave type with EC encryptor] > org.hyperledger.besu.tests.acceptance.privacy.PrivacyClusterAcceptanceTest.canInteractWithMultiplePrivacyGroups[TESSERA enclave type with EC encryptor] FAILED
    java.lang.RuntimeException: java.util.concurrent.TimeoutException: Tessera did not start
        at org.hyperledger.enclave.testutil.TesseraInternalProcessTestHarness.start(TesseraInternalProcessTestHarness.java:104)
        at org.hyperledger.besu.tests.acceptance.dsl.privacy.PrivacyNode.start(PrivacyNode.java:194)
        at org.hyperledger.besu.tests.acceptance.dsl.privacy.PrivacyCluster.startNode(PrivacyCluster.java:162)
        at org.hyperledger.besu.tests.acceptance.dsl.privacy.PrivacyCluster.startNode(PrivacyCluster.java:141)
        at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
        at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179)
        at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:992)
        at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
        at java.base/java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290)
        at java.base/java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:754)
        at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373)
        at java.base/java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:686)
        at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159)
        at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173)
        at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
        at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)
        at org.hyperledger.besu.tests.acceptance.dsl.privacy.PrivacyCluster.startNodes(PrivacyCluster.java:71)
        at org.hyperledger.besu.tests.acceptance.dsl.privacy.PrivacyCluster.start(PrivacyCluster.java:52)
        at org.hyperledger.besu.tests.acceptance.dsl.privacy.PrivacyCluster.start(PrivacyCluster.java:48)
        at org.hyperledger.besu.tests.acceptance.privacy.PrivacyClusterAcceptanceTest.<init>(PrivacyClusterAcceptanceTest.java:104)

        Caused by:
        java.util.concurrent.TimeoutException: Tessera did not start
            at org.hyperledger.enclave.testutil.TesseraInternalProcessTestHarness.startTessera(TesseraInternalProcessTestHarness.java:127)
            at org.hyperledger.enclave.testutil.TesseraInternalProcessTestHarness.start(TesseraInternalProcessTestHarness.java:98)
            ... 19 more

191 tests completed, 122 failed, 2 skipped

> Task :acceptance-tests:tests:acceptanceTestPrivacy FAILED

[macfarla]

looking good but still some flakiness in CI https://app.circleci.com/pipelines/github/hyperledger/besu/26379/workflows/a4201b08-0c73-4f2b-8c7e-05af8748847e/jobs/173575

PrivacyGroupAcceptanceTest > [TESSERA enclave type with EC encryptor] > org.hyperledger.besu.tests.acceptance.privacy.PrivacyGroupAcceptanceTest.canInteractWithMultiplePrivacyGroups[TESSERA enclave type with EC encryptor] FAILED
    java.lang.RuntimeException: java.util.concurrent.TimeoutException: Tessera did not start
        at org.hyperledger.enclave.testutil.TesseraInternalProcessTestHarness.start(TesseraInternalProcessTestHarness.java:104)
        at org.hyperledger.besu.tests.acceptance.dsl.privacy.PrivacyNode.start(PrivacyNode.java:194)
        at org.hyperledger.besu.tests.acceptance.dsl.privacy.PrivacyCluster.startNode(PrivacyCluster.java:162)
        at org.hyperledger.besu.tests.acceptance.dsl.privacy.PrivacyCluster.lambda$selectAndStartBootnode$6(PrivacyCluster.java:133)
        at java.base/java.util.Optional.ifPresent(Optional.java:178)
        at org.hyperledger.besu.tests.acceptance.dsl.privacy.PrivacyCluster.selectAndStartBootnode(PrivacyCluster.java:130)
        at org.hyperledger.besu.tests.acceptance.dsl.privacy.PrivacyCluster.startNodes(PrivacyCluster.java:67)
        at org.hyperledger.besu.tests.acceptance.dsl.privacy.PrivacyCluster.start(PrivacyCluster.java:52)
        at org.hyperledger.besu.tests.acceptance.dsl.privacy.PrivacyCluster.start(PrivacyCluster.java:48)
        at org.hyperledger.besu.tests.acceptance.privacy.PrivacyGroupAcceptanceTest.<init>(PrivacyGroupAcceptanceTest.java:89)

        Caused by:
        java.util.concurrent.TimeoutException: Tessera did not start
            at org.hyperledger.enclave.testutil.TesseraInternalProcessTestHarness.startTessera(TesseraInternalProcessTestHarness.java:127)
            at org.hyperledger.enclave.testutil.TesseraInternalProcessTestHarness.start(TesseraInternalProcessTestHarness.java:98)
            ... 9 more

50 tests completed, 2 failed

[macfarla]

@gtebrean it's failing locally for me because the tessera tar file doesn't exist at this location

  systemProperty 'tessera-dist', "${buildDir}/tessera/dist/tessera-${tesseraDist}/bin/tessera"

[macfarla]

@gtebrean I played around with circle CI config for the privacy ATs task today to see if I could make it more efficient - most recent run worked but took 43 min which isn't really feasible so I changed it back

[gtebrean]

@macfarla I tried to address the tx pool issue that we are facing, in the last commit. I followed the approach which was adopted in #6384 but I update it for the privacy node use case creating a new PrivateSyncCondition, using the PrivConditios. However the issue still persists locally and in github pipeline. Could you please look at my last commit? As I haven't interacted that much with synchronisation, maybe I missed something.

[macfarla]

I think your latest commit fixes the "Transaction pool not enabled" issue. (could you push commit this as a separate PR - I think we can merge that ahead of this one to fix the nightly). This PR though still has the occasional "Tessera did not start" which seems to be flaky for a different reason.

[macfarla]

Also - unit test failures is files missing headers

• What went wrong: Execution failed for task ':checkSpdxHeader'.

Files without headers: /home/circleci/project/acceptance-tests/dsl/src/main/java/org/hyperledger/besu/tests/acceptance/dsl/condition/priv/PrivateSyncingStatusCondition.java /home/circleci/project/acceptance-tests/dsl/src/main/java/org/hyperledger/besu/tests/acceptance/dsl/transaction/privacy/PrivSyncingTransactions.java

[macfarla]

@gtebrean for me it's still missing the part that will download the tessera dependency and install it into the build dir

[gtebrean]

@macfarla please mention in here the tasks you are trying to run to check exactly, in my case the tessera is added to the build folder. Sorry also for inactivity but I was really busy on projects. I tried to update this to latest changes, but BesuConfig class was updated and now all the privacy tests needs to be updated...at least all of them are failing when I run the job locally.

[macfarla]

@gtebrean I think we can close this PR, you have other methods to test privacy right, and we aren't going to spend time trying to get this working again with Tessera within Besu.