Add stress test with many indexes

[LeeKingHung]

Issue Basics

• ObjectBox version: 1.5.0
• Reproducibility: most of the times

Description

I found that when I have many data (>50,000 records) in the database, write operation ( Box.put(), Box.removeAll(), etc. ) will hang forever. However, read operation still works. After debugging, I found that "nativeBeginTx()" (in BoxStore.java) or "nativeDeleteAll" (in Cursor.java) method call never ended. Is there any additional setup for large data application?

My BoxStore.diagnose() result (called before write operation):

pid thread txnid 8875 b774fc00 - 8875 99c3a930 - 8875 a00ba930 - 8875 8c800930 -

entries=1988217 depth=4 branch-pages=466 leaf-pages=43826 overflow-pages=8 page-size=4096 last-page-number=48054 last-tx-id=3863 size=10737418240 max-readers=500 readers=4

Setup

static final int MAX_NUM_DB_READERS = 500;
static final long MAX_DB_SIZE_KB = 10 * 1024 * 1024;

mBoxStore = MyObjectBox.builder().androidContext( this ).maxSizeInKByte( MAX_DB_SIZE_KB ).maxReaders( MAX_NUM_DB_READERS ).build();

Debug

(Logged from the start of the application, where #4 is box.removeAll() transaction, The number of records in that box is < 300, and it takes forever) 

06-05 03:19:54.030 18382-18425 I/Box: TX #3 (read) ready on thread 9943a930
06-05 03:19:54.043 18382-18425 I/Box: TX #3 recycle
06-05 03:19:54.454 18382-18409 I/Box: TX #4 (write) ready on thread 9fb7f930
06-05 03:20:27.315 18382-18401 I/Box: TX #5 (read) ready on thread 9ffc0930
06-05 03:20:34.070 18382-18401 I/Box: TX #5 is destroyed on thread 9ffc0930...
06-05 03:20:34.070 18382-18401 I/Box: TX #5 destroyed
06-05 03:20:34.869 18382-18401 I/Box: TX #6 (read) ready on thread 9ffc0930
06-05 03:20:34.887 18382-18401 I/Box: TX #6 is destroyed on thread 9ffc0930...
06-05 03:20:34.887 18382-18401 I/Box: TX #6 destroyed
06-05 03:20:39.449 18382-18401 I/Box: TX #7 (write) ready on thread 9ffc0930
06-05 03:20:39.449 18382-18401 I/Box: TX #7 waiting for TX #4 to finish

[greenrobot]

50k entities is not much. We tested with millions.

Do you have some reproducible test case?

[LeeKingHung]

I do not have any test case yet as I am still in development.

I noticed that the number of entries is ~2M (From BoxStore.diagnose()), does it equal the number of records?

Anyway, I removed some indexes and relations from some entities, the database size was greatly reduced and no hang was observed.

[greenrobot]

Indexes influence the number of entries directly; 1 per index per entity. Out of curiosity: how many indexes where you having when it was slow? How many are you having now? PS.: Did it actually hang forever or was it very slow?

[LeeKingHung]

Previously I had 20++ indexes and now 10 indexes.

It hanged forever, i.e. no response at all (tried to wait for more than 10 minutes).

Maybe need to know what is the possible cause to make nativeBeginTx hang when data size is large?

[greenrobot]

There should be no reason to hang, just to slow down. We could set up a stress test on our side using a lot of index properties.