Truffle version from 5.1.49 to later versions slows down tests significantly

[hellwolf]

• [x] I've asked for help in the Truffle Gitter before filing this issue.

---

Issue

By upgrading from truffle version 5.1.49 to 5.1.65 slows down tests significantly

Steps to Reproduce

$ git clone https://github.com/superfluid-finance/protocol-monorepo
$ cd protocol-monorepo
$ yarn install
$ yarn build
$ cd packages/ethereum-contracts
$ npx truffle test test/contracts/agreements/ConstantFlowAgreementV1.test.js

Clock the time.

Now do the same thing after changing in package.json the truffle version to 5.1.65, then do yarn install, and test again.

Expected Behavior

Test speed should not deteriorate, if not improve.

Actual Results

Test speed slowed twice.

Environment

• Operating System: Ubuntu
• Ethereum client: ganache
• Truffle version (truffle version): 5.1.49, 5.1.65
• node version (node --version): v12.20.1
• npm version (npm --version): 6.14.11

[haltman-at]

Do you know what the first version that has this problem is? Thank you!

[hellwolf]

@haltman-at I think 5.1.49 was the last known "good" version for us.

[haltman-at]

So, are you saying that the issue was in fact introduced in 5.1.50, or is 5.1.49 simply the most recent version you happened to try that worked out?

[hellwolf]

@haltman-at

I tried again following the reproduce steps.

• with truffle 5.1.49, the test took 265.334 seconds.
• with truffle 5.1.50, the test took 357.748 seconds.

[haltman-at]

I see, OK, thanks! That gives us a place to start looking. Thanks for the report!

[haltman-at]

OK, so the most likely culprit for this seems to be that we upgraded web3 to 1.2.9...? It's not clear what we can do about this; we can't upgrade further at the moment due to various problems with later versions that haven't been fixed yet (specifically: this one from 1.2.10 to 1.3.3, and this one in 1.3.4). That said, this week's release is going to include what should be an across-the-board performance improvement (#3794), so hopefully that should help mitigate this!

[hellwolf]

okay, I will test again once it's released

[hellwolf]

the problem persists with 5.1.67

[haltman-at]

Troublesome. OK, we'll have to look into this more later. Thanks for alerting us.

[sirhill]

I've encountered what I think is the same issue.

Here before and after circle-ci builds:

Sub 10 minutes successfull build (11 Oct 2020)

https://app.circleci.com/pipelines/github/c-layer/contracts/139/workflows/f667581a-d186-44f3-8cce-5cb4df2fb52b

Over 20 minutes successfull build (22 Oct 2020)

https://app.circleci.com/pipelines/github/c-layer/contracts/158/workflows/da04c938-f6f5-4908-9cd4-d5ed66991921

Only minor changes appart.

[hellwolf]

Today I tried with "truffle": "5.2.3", the testing speed still experiences major slow down:

https://github.com/superfluid-finance/protocol-monorepo/actions/workflows/ci.yml

[hellwolf]

BUMP, I think it's quite high priority for us. Otherwise we will have to make the move to some other framework in the long run. We are already under lots of questioning why we are still using truffle. I always use truffle since I have been in this space long enough and the inertial has been the major factor while new comers always asking for using new tools. But I always believe truffle will be improving and still be the major framework going forward.

[haltman-at]

Is this improved any with Truffle 5.3.1? We think web3 was the source of the problem and we just upgraded that, so I'm curious whether it improved things at all.

[hellwolf]

@haltman-at thanks for coming back on this.

Unfortunately not, it still takes lots of time with 5.3.1. I am happy to do some benchmarking and traces to identify where is the hotspot or bottleneck for the slowdown. I could do some research on tools to be used, unless you have something to recommend me to us?

[Arvolear]

Encountered the same issue. I believe the problem is indeed in web3, cause truffle test runner and openzeppelin test runner are experiencing the same degradation by running a lot of tests in the project.

I have noticed that in ganache there are a lot of logs of eth_getBlockByNumber for each transaction in test execution (2-3k). I think that this is a consequence of transaction execution time slowing down for some reason while running a lot of tests (big file or all tests in a project).

Seems like the whole environment doesn't free up itself before each test. Running the test files separately speeds up the process a bit (because truffle hard resets).

[haltman-at]

@Arvolear, you may possibly want to use the disableConfirmationListener network configuration option to avoid the large number of eth_getBlockByNumbers.

Interesting that running the tests separately is faster. That's certainly worrying. We'll have to look into that, thanks.

[joequant]

We are also running into slow downs with 5.1.50 in which each test seems to be taking longer and longer.

[hellwolf]

It should have a higher priority.

[Arvolear]

Absolutely agree, we were forced to downgrade to 5.1.49 because the later versions are pretty much unusable.

[eggplantzzz]

@Arvolear did you try @haltman-at's suggestion to use disableConfirmationListener? web3 polls for confirmations if this is enabled which can be expensive.

[Arvolear]

@eggplantzzz Hey, I tried and disabled the listener, however, the major truffle version still experienced a huge speed degradation while running the tests in the project (there are roughly 400 of them in 20 separate test files).

While using the 5.2.6 version (back then) the CI would take more than 2 hours to complete and downgrading truffle to 5.1.49 reduced the time to 10 minutes.

I am still running the tests with the disableConfirmationListener and from my personal analysis, the tests get faster by 10-15% with it.

[eggplantzzz]

Hmm that is not good, that is still quite a bit slower. Can you link a repo that we can use to try and reproduce your issues?

[devinaconley]

I can also confirm that this affected the open zeppelin test runner. We were experiencing 2-3x slowdowns, especially when running a large number of tests.

Reverting to these versions resolved the issue

    "@openzeppelin/test-environment": "^0.1.4",
    "@openzeppelin/test-helpers": "^0.5.6",
    "truffle": "^5.1.39",
    "web3": "^1.2.11",
    "web3-utils": "^1.3.4"

I didn't exhaustively check for the "breaking" version, so there may be working ones in between

[hellwolf]

guys, this is getting really annoying, anything we can help with?

[hellwolf]

Disabling disableConfirmationListener definitely helped!

But how do you configure the default "test" network, the one that is launched as a "internal ganache" by truffle test?

[haltman-at]

guys, this is getting really annoying, anything we can help with?

Unfortunately this may be difficult. As best we can tell, the cause of this issue is due to a web3 upgrade. This is by process of elimination; none of the other changes made in v5.1.50 are a plausible candidate. But we're not about to downgrade web3, so...?

Now if we're wrong and it's due to one of the other changes in that release, then there might be more to work with here, but at the moment there's no obvious way forward.

[haltman-at]

Disabling disableConfirmationListener definitely helped!

Glad to hear it!

But how do you configure the default "test" network, the one that is launched as a "internal ganache" by truffle test?

Good question. You can of course configure the test network... but if you do so, Truffle Test will look for that network to already exist (i.e. you'll have to start up Ganache separately), rather than creating its own internal Ganache.

So the answer to your question is that at the moment, there is no way to do that. We ought to change that. I'll go file an issue for it!

[vyomshm]

Truffle tests are so slow. Please increase the priority for this!

[rchrdw]

I'm experiencing this issue as well.

For a very simple project, tests on Truffle v5.5.6 took 23s, Truffle v5.1.49 took 1s.

[eggplantzzz]

Hey @rchrdw, are you using an http provider? If you use a websocket provider in your test network configuration does it speed them up?