Can't get BTree data to persist

[colin-infersens]

Hi,

I'm developing an application in MicroPython on an ESP32 where a buffer of data to be published needs to persist over power outages/reboots.

BTree seems like the obvious solution, but in my tests I can't get the data written to the BTree db to persist - in fact it doesn't seem to be written to the file at all.

Here is some sample test code; I'd be really grateful if anyone can offer any assistance, please:

import btree

try:
    f = open("buffer.db", "r+b")
except OSError:
    f = open("buffer.db", "w+b")
db = btree.open(f)

db[b"1"] = b"first"
db[b"2"] = b"second"
db[b"3"] = b"third"
db[b"4"] = b"fourth"
db[b"5"] = b"fifth"

db.flush()

print('The database contains the following:')

for k in db:
    key = k.decode('utf-8')
    val = db[k].decode('utf-8')
    print('Key:', key, 'Value:', val)

print('End of data.')

db.close()
f.close()

This all appears to work fine, with the expected output to REPL:

The database contains the following:
Key: 1 Value: first
Key: 2 Value: second
Key: 3 Value: third
Key: 4 Value: fourth
Key: 5 Value: fifth
End of data.

But if I then try the following:

f = open("buffer.db", "r+b")
db = btree.open(f)

print('The database contains the following:')

for k in db:
    key = k.decode('utf-8')
    val = db[k].decode('utf-8')
    print('Key:', key, 'Value:', val)

print('End of data.')

db.close()
f.close()

...the output to REPL is:

The database contains the following:
End of data.

On checking the db file "buffer.db", it has a size of 0 bytes.

I'm sure I'm making a schoolboy error, but I can't spot it for the life of me.

Many thanks in advance for your help.

[colin-infersens]

I'm closing this issue, as I've duplicated it under micropython/micropython issue 4327 where I thought it would be more appropriate.

[colin-infersens]

@loboris - this issue appears to be specific to this branch, so I've re-opened it here.

With some testing on BTree in the micropython/micropython repo on ESP32 (please see here ), it appears to work as expected, but when using the same test code on the same device using this repo, it doesn't work, as described in the original post, above.

[loboris]

Thank you for reporting this.

It looks there are some issues when running BTree on spiffs file system. As spiffs is the default file system type, you are probably using it. I'll try to find the cause of the issue and fix it.

If the file system is formated as FatFS, BTree can be used without issues. Select FatFS FS type in menuconfig: → MicroPython → File systems → Set internal filesystem type (FatFS)

BTW, in the next update (probably next week) full support for SQlite3 database will be included.

[colin-infersens]

OK, thanks for the feedback @loboris - I'll do some testing with the filesystem formatted as FatFS, but looking forward to SQLite3 in the next update.