Additional mempool API endpoints

[arobsn]

Keeping the mempool synced with an external service (like an explorer, for example) is not very effective and such an overhead for many reasons. So as mempool is a relativelly small in-memory data set, I think that it would be really good to be able to query the mempool directly in the node. Also, it can potentially make possible for explorers' APIs to reflect mempool in realtime –which hopefully will be the case for ergo-uexplorer`s GraphQL server.

Here I'm listing a set of endpoints that should be enough for a wide variety of use cases.

• Transactions

  • HEAD: mempool/transactions/{transactionId} - Check if a transaction if present in mempool
  • GET: mempool/transactions/byTransactionId/{transactionId} - Get a transaction by transactionId
  • GET: mempool/transactions/byErgoTree/{ergoTree} - Get a list of transactions filtered by ergoTree, considering inputs and outputs

• Inputs

  • POST: mempool/transactions/inputs/?check={true|false} - Batch get a list of inputs filtered by multiple boxIds. If check=true it must check if boxIds list elements are present on mempool as inputs and only return a list of boxId of included ones.
  • GET: mempool/transactions/inputs/byBoxId/{boxId} - Get an input by boxId;

• Outputs

  • POST: mempool/transactions/outputs/?check={true|false} - Batch get a list of outputs filtered by multiple boxIds. If check=true it must check if boxIds list elements are present on mempool as outputs and only return a list of boxId of included ones.
  • GET: mempool/transactions/outputs/byBoxId/{boxId} - Get a box by boxId
  • GET: mempool/transactions/outputs/byErgoTree/{ergoTree} - Get a list of boxes filtered by ergoTree
  • GET: mempool/transactions/outputs/byTokenId/{tokenId} - Get a list of boxes filtered by tokenId
  • POST: mempool/transactions/outputs/byRegisters - Get a list of boxes filtered by a set of hex registers. It must filter only by defined registers and not consider undefined ones.

    // proposed registers filter param 
    {
      R4: string | undefined,
      R5: string | undefined,
      R6: string | undefined,
      R7: string | undefined,
      R8: string | undefined,
      R9: string | undefined
    }
    

[MrStahlfelge]

I agree the mempool API should be a bit more capable to avoid fetching it completely. At the moment, wallet app and all appkit clients fetch it completely to build chained transactions.

  * `HEAD`: `mempool/transactions/{transactionId}` - Check if a transaction if present in mempool
  * `GET`: `mempool/transactions/byTransactionId/{transactionId}` - Get a transaction by `transactionId`

Agreed, great to check if a tx is waiting

  * `GET`: `mempool/transactions/byErgoTree/{ergoTree}` - Get a list of transactions filtered by `ergoTree`, considering inputs and outputs

Agreed, great to filter by address

  * `GET`: `mempool/transactions/inputs/byBoxId/{boxId}` - Get an input by `boxId`;

What is the point here to need this over getBoxById?

  * `GET`: `mempool/transactions/outputs/byBoxId/{boxId}` - Get a box by `boxId`

What is the pointer here over getBoxByIdWithPool?

Given that the mempool is very small (<= 100 tx), it feels like the dedicated filtering endpoints are not all necessary and will make the node unnecessary complex.

[pragmaxim]

  * `GET`: `mempool/transactions/byErgoTree/{ergoTree}` - Get a list of transactions filtered by `ergoTree`, considering inputs and outputs

ErgoTree/Address can be too big to fit into GET ... so this must be POST

If you want to see the real sizes, check explorer-backend or random blocks from node, ergoTree/address can overflow 2000 char http limit for url :

    ergo_tree               VARCHAR, // hex of ergo tree
    ergo_tree_template_hash VARCHAR, // Sha256 encoded hex of ergo tree tempate (Serialized proposition expression of SigmaProp type with ConstantPlaceholder nodes instead of Constant nodes  )

[pragmaxim]

  * `HEAD`: `mempool/transactions/{transactionId}` - Check if a transaction if present in mempool
  * `GET`: `mempool/transactions/byTransactionId/{transactionId}` - Get a transaction by `transactionId`

I'm not a REST expert but isn't the rule of thumb here to have only one GET endpoint that returns 200 or 404 Not Found ?

* `GET`: `mempool/transactions/outputs/byBoxId/{boxId}` - Get a box by `boxId`

I assume this would be like getBoxByIdWithPool but for mempool only, right ?

  * `GET`: `mempool/transactions/inputs/byBoxId/{boxId}` - Get an input by `boxId`;
What is the point here to need this over getBoxById?

@MrStahlfelge ^^^ I guess this is for returning UTXOs used by unconfirmed transactions, sort of makes sense

POST: mempool/transactions/inputs/?check={true|false}
POST: mempool/transactions/outputs/?check={true|false}

@anon-br ^^^ these seem cumbersome, I guess the dApp iterating over /transactions/inputs/byBoxId/{boxId} and /transactions/outputs/byBoxId/{boxId} doing the logic themselves is less work than grasping this endpoint ))

GET: mempool/transactions/outputs/byBoxId/{boxId} - Get a box by boxId
GET: mempool/transactions/outputs/byErgoTree/{ergoTree} - Get a list of boxes filtered by ergoTree
GET: mempool/transactions/outputs/byTokenId/{tokenId} - Get a list of boxes filtered by tokenId

^^^^ These seem to be easy to implement and understand

POST: mempool/transactions/outputs/byRegisters - Get a list of boxes filtered by a set of hex registers. It must filter only by defined registers and not consider undefined ones.

@anon-br This means the Body would contain {R6 : x, R7 : y} and we would return only boxes that contain these 2 registers with corresponding values? I guess this make sense

[pragmaxim]

Overall, I agree with all except for these as they are cumbersome and can be easily worked around :

HEAD: mempool/transactions/{transactionId}
POST: mempool/transactions/inputs/?check={true|false} 
POST: mempool/transactions/outputs/?check={true|false}

As I'm sure that dApps often need to differentiate between confirmed & unconfirmed

[arobsn]

Imho it should not be a problem for any dApp to call Base16.encode(ergoTree) before issuing that request so we should probably avoid passing around huge ErgoTree strings. WDYT?

I agree, should not be a problem.  

* `HEAD`: `mempool/transactions/{transactionId}` - Check if a transaction if present in mempool
* `GET`: `mempool/transactions/byTransactionId/{transactionId}` - Get a transaction by `transactionId`

I'm not a REST expert but isn't the rule of thumb here to have only one GET endpoint that returns 200 or 404 Not Found ?

I'm not aware of the REST API rules on the node, but the HEAD request is very useful if a dApp wants to check if a transaction is in mempool without needing to fetch the whole transaction data for it.  

* `GET`: `mempool/transactions/outputs/byBoxId/{boxId}` - Get a box by `boxId`

I assume this would be like getBoxByIdWithPool but for mempool only, right ?

Right!  

POST: mempool/transactions/inputs/?check={true|false}
POST: mempool/transactions/outputs/?check={true|false}

@anon-br ^^^ these seem cumbersome, I guess the dApp iterating over /transactions/inputs/byBoxId/{boxId} and /transactions/outputs/byBoxId/{boxId} doing the logic themselves is less work than grasping this endpoint ))

Batch requests are really useful for optimizations (specifically for wallets) and on avoiding multiple calls at once on the same endpoint. However it's something just good to have, not mandatory. So if it will hurt API design, better to not support it at all.  

POST: mempool/transactions/outputs/byRegisters - Get a list of boxes filtered by a set of hex registers. It must filter only by defined registers and not consider undefined ones.

@anon-br This means the Body would contain {R6 : x, R7 : y} and we would return only boxes that contain these 2 registers with corresponding values? I guess this make sense

Exactly this.  

[ghost]

Issue looks closable (#1869 contains Closes #1854, sometimes auto-close does not act)

[pragmaxim]

Yeah, this was merged into 4.0.105

[ghost]

Bounty was not paid though.

Payment should be executed on successful merge.

Is there any documentation about the bounty process?

[ghost]

Is there any documentation about the bounty process?

https://github.com/ergoplatform/grow-ergo/

process is not described.

[ghost]

@kushti