Enable Box Storage

[algoanne]

See epic for context.

Solution

(some details are TBD)

Applications should be able to use box storage, which works in the following way:

• Boxes must be allocated explicitly by the app.
  • The size of the box is defined when allocating it. The size will have a maximum, likely around 16 or 32k
  • The box is named by a byte string, in an app private namespace. The box name can have a max size of 64. (same as global/local key names)
  • An app can allocate any number of boxes.
  • The app account’s minimum balance requirement (MBR) is increased by 2500 per box + 400 per byte (tentative)
• Box sizes and names cannot be changed after initial allocation. (Though as box could be deleted and recreated.)
• Values stored in boxes are stored and accessed as bytes. Opcodes all accessing/updating ranges of bytes from box.
• Boxes are only visible to the app itself: other applications cannot view the app’s boxes on-chain (of course, everything can be seen from off-chain, for example by using the algod API).
  • Note: the re-entrancy restriction means that if app A calls app B, app B will not be able to view app A’s box storage (cannot call app A’s functions to view it). So A should pass important info.
• For an app A to access its boxes, the box names must be pre-specified in the txn call (this is analogous to foreign arrays).
  • The number of boxes that can be used in one call will be limited by the total size of these boxes.
• An application can be deleted while it has outstanding boxes. In this situation, its MBR is un-recoverable from the deleted app account. (Just as an app account's balance is burned forever if app deleted.)
• Specific opcodes are TBD, but will include something like: create, extract (byte range), replace (byte range), delete.

This ticket includes the work to update goal app to add a --box flag so that boxes can be specified when calling an app.

---

Itemizing pending tasks:

• [x] Merge 320 round support (via https://github.com/algorand/go-algorand/pull/4255).
• [x] Support kvstore table migration during consensus upgrade (via https://github.com/algorand/go-algorand/pull/4229).
• [ ] Support catchpoints.
• [ ] Carryover fix for lookupResource error handling.
• [ ] Configure baseKVPendingBufferSize based on experimentation.

[jeapostrophe]

• An application can be deleted while it has outstanding boxes. In this situation, its MBR is un-recoverable from the deleted app account. (Just as an app account's balance is burned forever if app deleted.)

I want to beg you to reconsider this.

This is not how local storage works. If a user allocates space in application 42, then when it is delete, the user can restore their storage by issuing a ClearState that runs even though there is no longer a ClearStateProgram to run. This is a special code path that is supported that allows users to reclaim their MBR.

While it is true that app account balances are lost forever, it is very easy to avoid this by having the DeleteApplication txn generate a close out to the application creator. This is a very small amount of extra code for ALGOs (although it is a bit tedious for ASAs, because you have to remember all of them and close them out individually.) Currently, Reach generates that automatically for all contracts.

I think it is significantly more difficult to handle freeing box MBRs correctly. Essentially you need to "end" the application and then go into a new mode where the only kind of a txn you accept is like a deleteBox call that deletes the boxes one-by-one. I will easily be able to integrate this into Reach, but I think it will impose a space tax on programs and, worse, it means that we have to build some service to go and find applications with boxes that can be freed and free them. There are weird economic questions here as well... does the MBR go to the freer? The creator? Both? A split?

I think that most people will not do this at all or at least not correctly and thus you are opening a path to destroying a lot of the ALGOs that exist through lost boxes. Now, maybe you're comfortable with that because people can already launch contracts that lock their application account funds away or have accrued rewards and will never harvest them and so on, but I am very anxious about these issues.

My ideal is that when an application gets delete, the entire MBR is freed and returned to the creator. Individual applications can implement different policies by not deleting themselves but doing something else. I believe that you don't think this is possible because it is non-linear work on the ledger to free the space and return the MBR. I believe in you to think of a solution to that.

My second best option is that if an application ends with boxes committed to it, then anyone can free a box and take the MBR back. This incentivizes people to find them and yet it allows individual applications to implement different policies by having their own code. I believe that you don't want to do this because it would require a new kind of a transaction and that is expensive and annoying. I think you can make a new OnCompletion type called DeleteABoxPlease that is always rejected by non-deleted applications and on deleted applications, reads the key from ApplicationArg 0, deletes it, and returns the MBR to the caller.

My third best option is to implement the "kind of deleted" phase and pay the cost in space and complexity to do a good deed to the network. I will implement this if you don't do the others.

FWIW, application accounts opt-in to assets presents exactly the same problem. Currently Reach solves it by bounding the number to a static number per contract and enforcing that all are opted out in the final txns, which means that a contract can only opt in to about 256 (because of all the opt-out calls). If I have to do option three for boxes, then I'll do the same for assets. If I can do options one or two for boxes, then I'll try to think of something similar for assets. Assets are more complicated because they are controlled by the application account. One thing I've thought of is systematically rekeying the application account on deletion to the creator and providing a tool for harvesting its assets. Another fun option would be "unlocking" an account by "rekeying" it to anyone, so that people will go find unlocked accounts and recover their assets and MBRs. This is a general mechanism for these kinds of problems and would be a strategy for implementing my second-best-option... basically you could make a BoxDelete txn and think about that as something that the application account can issue. If I want to make it so that anyone (or a particular someone) can do that after the application is over, then I rekey to them and then they can start issuing BoxDelete txns to restore everything and harvest the MBRs.

Basically, I think this little line here is a really big deal and we should not accept second best.

[jannotti]

The "kind of deleted" phase is what I expected applications that care about their money to do. It's exactly what they have to do to recover MBR for opted-in assets or apps (or, more similarly, for ASAs and apps they create themselves). We've considered your second idea (users can scavenge deleted apps' boxes) and I don't hate it, but I have come to see this storage as "bought and paid for". An app has decided to scribble forever into the blockchain. It would be strange to take that right away - it's possible in so many ways. (However, the reason why I'm at least amenable to it is that if they really want to scribble forever, they could still update to "int 0" instead of deleting.

Note that you can not close out an account with any MBR, so, just as for any ASAs, you would have the "it is a bit tedious for ASAs, because you have to remember all of them and close them out individually" situation, but it would be similarly "safe". Presumably, you feel exactly the same about ASAs that an app account has opted into (or, even more complicated, apps and asas that the app account has created.)

I would be most agreeable to a solution that treats all the things an app account can create (ASAs, Apps, and Boxes) the same way. Right now, the proposed box behavior is consistent with those. I would not expect (but I could be wrong!) that you'd want an ASA that was created by an app account to disappear just because the app was deleted, any more that you'd want an ASA created by a person to disappear just because they lost their private keys. I think of app deletion and loss of keys as almost perfectly analogous. An account exists for which the authority is gone.

[jeapostrophe]

I agree with you on the third paragraph a lot. That's my point about the rekey-ing thing. Right now, I could rekey the application account and deal with the ASAs but not with the boxes. I think it would be nice to have that uniform system. I don't think this should block the release of boxes, but I don't want to declare boxes done forever with no opportunity to improve :)

[jannotti]

That's very good point about a human account being unable to deal with the problem "later". That needs some thinking. There's almost a workaround. You could write a new app, and rekey the old app account to it. But even that would not let the new app issue box_delete ops against the original boxes.

[michaeldiamant]

For ease of reference, integration branch PR: https://github.com/algorand/go-algorand/pull/4149.