Excessive wallet size causes performance and connectivity issues

[ADumaine]

Describe the issue

The wallet on the dev tracking server grew to over 1.4GB due to challenge transactions(~17k) and became an issue when looking up transactions. It was also taking a long time to load on restart and timing out on rpc calls. The wallet had to be replaced but is grower even faster due to additional new nodes. It is estimated the wallet will need to be replace every 10-14 days or sooner at 600 with 1 challenge a day.

Additional detail

Can you reliably reproduce the issue?

Yes by allowing the server's wallet to grow without replacing.

Expected behaviour

Fast loading on restarts, no time outs on rpc calls, fast transaction lookup in the shielded address.

The actual transaction content would not be needed after a challenge is closed.

Suggested solution

@aayanl suggested: "We can probably add a flag to start dropping transactions after a certain number Like -retaintransactions=100 to only keep the last 100 transactions in the wallet.dat"

This may be an ideal solution for the tracking servers. No work-around would be needed to replace wallets periodically. This is a potential stumbling block for production deployment.

[ADumaine]

I have the large wallet mentioned above available if it is needed for testing. 200MB zipped.

[WaveringAna]

related issue https://github.com/zcash/zcash/issues/1749