objectbox-java
objectbox-java copied to clipboard
objectbox
Option to shrink unused space
Did not find any info on that, so decided to ask here - is there (or planned) an option to shrink (e.g. compact the database)?
We're using objectbox for quite a long time for a production app one entity there is used to collect various loggable information for further sending it to a server. Successfully sent entities gets removed from the database, however, the physical db file still grows constantly. (which is understandable)
I understand we can always delete the file before creating the ObjectBox instance in order to reduce the unused space, however there are other entities there so we'll need save and recreate them somehow in the new (emptied) datastore. Thus, kind of an option to the builder like .shrinkOnOpen() would be nice to have, so app may check the database size during the initialization process and shrink it when necessary.
Thanks!
I placed the entities which are most frequently deleted in a separate BoxStore and when this reaches a certain size, the entities in it are sent to server and the BoxStore is recreated by calling deleteAllFiles() and then rebuilding it. Additionally the entities that need to be retained could be read and kept in RAM (provided it is big enough) while the BoxStore is being shrunk and then written again to the new tighter store.
I just want to make sure that the DB size is growing within expected bounds: basically, if you remove and put comparable sized entities in the same quantities, the required space should not grow anymore at a certain point. Is that what you are seeing, too?
Yes, indeed, the size of the BoxStore is stable, when some entities are deleted and new ones are saved the size remains constant, so there is absolutely no problem with the database. The only thing is the BoxStore file is not trimmed after deleting entities which is perfectly normal from a systems' point of view since it will grow right back afterwards and it will add unneeded complexity and overhead to the engine.
All my fiddling with the BoxStore was due to a business requirement that the size of the app storage must not exceed an arbitrary limit and in case that limit is reached no more entities should be created. Since the usage patterns of the application are quite diverse in the few cases that limit was reached the data was removed from the BoxStore but its size remained the same, so the app would still not create new entities. That was the only reason for deleting the file and recreating it, no 'leaks' or 'fragmentation' as far as I could observe, the database works flawlessly and I hardly even notice it. Perhaps there is a simpler way to answer the request by simply exposing some low level attributes of the BoxStore like overall byte size and used byte size or loading factor (%).
In the end thank you very much for this awesome library which is truly a game changer.
ObjectBox will automatically use the free space inside the database after removing some entities? I understand this correctly? I used very large entities with binary data and my database grew very quickly, after removing some entities from the database, I added more new entities to exactly repeat those I deleted. I expected the new entities to use the remote place, but my base has become even more
Reuse of space previously occupied by deleted entities is more of an observation, not an advertised feature of the platform and it's probably a best effort attempt, not a guarantee. Moreover this observation was made without considering binary data. My guess is that storage space for binary fields (byte[]) is shared for several/all entities, so it may be difficult to anticipate if the available space can be reused at all because it might get fragmented by insertion of other entities which even if they use only a fraction of the overall space would still make it insufficient for the re-inserted entity that had been deleted in the beginning. Also are you sure the new entity is exactly the same as the deleted one? You might read the same data but the array holding it could be larger.
ObjectBox will automatically use the free space inside the database after removing some entities? I understand this correctly? I used very large entities with binary data and my database grew very quickly, after removing some entities from the database, I added more new entities to exactly repeat those I deleted. I expected the new entities to use the remote place, but my base has become even more
@crispert Thanks for the response. It's mostly correct. Space is reclaimed, but the behavior is too complex to predict (b-tree, pages...).
binary fields (byte[]) is shared for several/all entities
There no such thing, each object is stored independently from each other.
Do we have a way to shrink the database now? My app currently write data worth an MB every 2-3 seconds and after a few minutes, it sends the data to the server and deletes everything from the database. However, app size does not reduce at all.
it sends the data to the server and deletes everything from the database
In that case you could delete the DB completely instead:
boxStore.close();
boxStore.deleteAllFiles();
Thanks but actually there is a concurrency issue with that for me as while I am deleting all of the data that got uploaded to the server, new data is still getting stored in the database. So just deleting the database would result in loss of this new data and also, while I am deleting and recreating the database, new data cant get stored because the database is closed.
SOLUTION: I did end up using your advice but with an additional workaround. What I did was, I created a secondary database for temporary storage. So, while the database is getting deleted and recreated, I store every new data into this secondary storage and as soon as the recreation is done, I migrate all of the data back into my primary database and then recreate this secondary database as well to reclaim space occupied by it.
I am also having concerns about disk usage. How can one create a secondary database within the same app?
@LeoK987 You can build several instances of BoxStore which represents a database in Objectbox terms:
MyObjectBox.builder()
.androidContext(appContext)
.name("your_store_name")
.build();
Just be careful to give a unique name to each one. You can insert any entity in any store but then you'll have to query each store when you need the entities. And it would be helpful for the entities to have an intrinsic unique key as one entity might have different ids within each store.
I just want to make sure that the DB size is growing within expected bounds: basically, if you remove and put comparable sized entities in the same quantities, the required space should not grow anymore at a certain point. Is that what you are seeing, too?
This doesn't appear true. I noticed when I removed half of my data and added back 1/3 of the removed and the DB size actually grew from 750MB to 770MB. OB version 2.9.0
Likely you have not reached "a certain point". If you carefully read the FAQ, you will find some reasons as for why. E.g. how big transactions are have a influence. If you are interested in this, you need to dig a bit deeper and do some systematic testing before making conclusions.
Sorry, not making conclusions, just thought to present some observations if that could help you make conclusions.
Just did a total transfer of data onto a secondary DB and the file size gets to 280MB, a huge reduction from 770MB. The primary DB was a result of usage and development in the past months when data have been removed/added at times and models have been modified. I don't remember any spike of data imports in the past. The secondary DB is a pure full transfer from the primary DB and does not involve any data removal or model change.
I understand that in your case a DB shrink would save a huge amount disk space. I just want to point out that this is very specific to whatever you did with the database and can not be generalized for other use cases.
If only I know what in my specific usage causes disk space not being released. Note I have no other way of using the DB aside from what are allowed by the API.
Possibly two things of suspect (just speculating here as I don't know the internal work of the DB).
I had lists of string ids in models used for referencing related items. Then after I implemented ToMany relations, I removed those lists from the models without first setting them to null.
Then I found building the ToMany relations required too much memory and CPU resources that I can't afford, so I removed the ToMany relations from the models and re-instituted my lists of string ids. I can't say for sure if I had set the ToMany relations to null before removing them from the models. I might have done it on a test device but not on the primary device.
So, could these cause OB to not recognize the space as free?
I think without this feature, the objectbox cannot be used for any serious work.
Yes this is a really important feature for us
Scenarios: Because of some bug, It duplicated or stored too much unnecessary data now We can not do anything about it It will claim storage space forever as I am not going to store that much data ever again.
I am facing this issue in our app and need a solution for this (It is 1 GB of claimed storage)
Thank you.
@bibek-ranjan-saha As suggested in the comments above, a workaround is to create a differently named database. Then transfer data and delete the old database files. Or if it's just a cache, delete all database files before creating a store to start fresh.
FYI: internally, we have the basic functionality in-place to "defragment" a database (e.g. reclaim free pages) by making a fresh copy. However, this still must be integrated with opening the store (e.g. add options)... Just wanted to give some outlook...