Isolated series of queries via Client Query API

[0x009922]

Use case

I want to make a snapshot of Iroha state using a series of queries. The snapshot should be consistent, i.e. all queries should be done in an isolated way, without changes to the state while the queries are performed.

Desirable solution

Add a way to execute a series of queries on an isolated state via Client Query API.

Current solution

I guess it is currently possible to do via smartcontracts: I can create a manually-executed trigger that will perform all needed queries on a single state and emit any kind of aggregated output as an event with custom JSON payload.

[s8sato]

What are the use cases?

Why not adopt a dedicated, single query or endpoint that covers the entire state?

[0x009922]

I have outlined the use case in the description: to be able to make multiple different queries while being sure their data is consistent with each other. I.e. they are all made on the same StateView. For example, if I create a SQL snapshot of domains/accounts/assets, I need to fetch them "all at once" so that the entities reflect a consistent state and there are no e.g. foreign key violations.

It would also be okay to make a single multi-query request for this use case.

[s8sato]

Can I interpret this as combining the currently separate queries—FindDomains, FindAccounts, FindAssets, etc.—and executing them all at once against impl StateReadOnly, effectively providing functionality equivalent to an RDB JOIN?

Alternatively, would another solution be to allow creating snapshots not only from State (i.e. StateView), but also from StateBlock and StateTransaction?

[0x009922]

Can I interpret this as combining the currently separate queries—FindDomains, FindAccounts, FindAssets, etc.—and executing them all at once against impl StateReadOnly,

Yes, I mean executing separate queries against the same impl StateReadOnly.

effectively providing functionality equivalent to an RDB JOIN?

Mm, I see the similarity, but it is not equivalent. JOIN usually is useful to join data by some parameters server-side, which is not the case here. Maybe, full OUTER JOIN is more appropriate equivalence.

The exact equivalent to SQL would be using transactions to make multiple queries:

BEGIN;
SELECT ...;
SELECT ...;
SELECT ...;
ROLLBACK;
-- no changes made, just rollback

Alternatively, would another solution be to allow creating snapshots not only from State (i.e. StateView), but also from StateBlock and StateTransaction?

I don't see why use StateBlock or StateTransaction for queries for client API. But for queries made from within smartcontracts, yes, I guess they have to use StateTransaction.

[s8sato]

As long as the queries are based on the same StateView, no state changes can be interleaved and it should be safe to merge their results—so is that not currently possible?

[0x009922]

It is currently possible, right. But not from the client side.

[s8sato]

Finally, I understand the issue. Indeed, there is no guarantee that queries issued by the client are based on the same StateView. Perhaps the node could maintain a snapshot at the block height for as long as the client intends to continue querying?

[0x009922]

Perhaps the node could maintain a snapshot at the block height for as long as the client intends to continue querying?

I think it could be done by extending the existing "live queries" layer of Iroha. Currently its purpose is to hold intermediate state of iterable queries, so that clients could retrieve consistent batches from a single StateView over an extended period of time. However, it only works for a single query and the state is released once the iteration completes. I think it's possible to add a mechanism to maintain StateViews with similar rules (time limit, user limit) to allow make multiple queries with it.

A design attempt:

• New data model entities: enum QueryPersistence { Begin, Continue(QueryPersistenceKey) }, struct QueryPersistenceKey(String)
• Extend: QueryParams with persist: Option<QueryPersistence>, QueryResponse with persist: Option<QueryPersistenceKey>
• If client specifies persist: Some(QueryPersistence::Begin), Iroha will issue a unique QueryPersistenceKey and hold a StateView for it. It will respond with persist: Some(QueryPersistenceKey).
• Client then could specify persist: Some(QueryPersistence::Continue(key)) on subsequent POST /query requests.
• Iroha guarantees that all queries made with the same persistence key reflect a single state view.
• Iroha rejects queries with unknown/expired persistence keys.
• Persistence does not affect continuation of iterable queries, they have their own lifecycle anyway.
  • However, continuing iterable queries which are part of some greater persistence must prolong the life of the persistence.

[s8sato]

It seems that archival nodes typically maintain a large number of snapshots proactively, rather than creating them reactively in response to queries.