Limitations of indexer-related RPCs

[driftluo]

Goal or context for the feature

Previously, due to the large amount of data, some queries were already experiencing excessively long processing times. The most directly affected APIs were get_cells, get_transactions, and get_cells_capacity.

While the two PRs(https://github.com/nervosnetwork/ckb/pull/4576, https://github.com/nervosnetwork/ckb/pull/4529) introduced configurable limits for get_transactions and get_cells_capacity, get_cells did not improve this. It does not support request limit restrictions, and its problem lies in the fact that it may need to iterate through a large number of cells with a small return value. This is not entirely consistent with the issues of the other two RPC APIs. Furthermore, users can construct an empty filter to iterate through all cells, resulting in significant resource waste and potentially enabling DoS attacks.

relations：https://github.com/nervosnetwork/ckb/pull/4469

Describe the solution you'd like

The cost of these interfaces varies depending on the host machine's configuration. We plan to impose some time consumption limits on the interfaces, roughly as shown below:

struct TimeoutIterator<I> {
    inner: I,
    start_time: Instant,
    timeout: Duration,
    timed_out: bool,
}

impl<I: Iterator> Iterator for TimeoutIterator<I> {
    type Item = I::Item;

    fn next(&mut self) -> Option<Self::Item> {
        if self.start_time.elapsed() > self.timeout {
            self.timed_out = true;
            return None;
        }
        self.inner.next()
    }
}

During the iteration, time will be controlled. A single query will have a default maximum time interval of 5-10 seconds. If a timeout occurs, the query will be forcibly terminated, and an error will be returned.

Perhaps nginx should also be equipped with corresponding interface query timeout configurations, or provide a similar template with default configurations and timeout configurations for whether RPC interfaces are public or private.

cc：@chenyukang @phroi

[quake]

Returning None when a timeout occurs will cause the outer code to interpret the iteration as having completed normally. This can lead to incomplete results being processed or reported. To properly signal that a timeout has occurred, the iterator should return a Result<Item, TimeoutError> instead, so that the caller can distinguish between a normal end of iteration and an early termination due to a timeout.

[quake]

one timeout field should be enough:

struct TimeoutIterator<I> {
    inner: I,
    timeout: Instant,
}

fn next(&mut self) -> Option<Self::Item> {
        if Instant::now() > self.timeout{
            return Some(Err(TimeoutError));
        }

[driftluo]

one timeout field should be enough

The time_out: bool field is used to indicate whether the process terminated normally or after a timeout. Returning a Result would complicate the iterator's processing. Adding this field allows the original processing logic to remain unchanged, with the result confirmed only at the end by checking this field.

like this：https://github.com/nervosnetwork/ckb/pull/5012/files#diff-1be6d84bf0230c2b83df27c01047bef048e84edf06c7f8bcbf9da81796522529R372-R376

[quake]

you may using iter.map(|res| res?) to adapt an iterator that returns Result<Item, Error> into a form that can be used with standard iterator adapters like take_while or filter_map. The main advantage is that it minimizes code changes, you don’t need to add extra checking to the iterator.

[driftluo]

you may using iter.map(|res| res?) to adapt an iterator that returns Result<Item, Error> into a form

Internally, the iterator cannot directly return to the function error. If we directly use is_ok to remove the Result, then there's no way to determine whether it's a timeout or a normal return.

Either the iterator syntax needs to be changed to for item in iter, or try_fold needs to be used to integrate error handling, take_while, filter_map, and take. Either way, the changes will be more significant than they are now.

[quake]

you may using iter.map(|res| res?) to adapt an iterator that returns Result<Item, Error> into a form

Internally, the iterator cannot directly return to the function error. If we directly use is_ok to remove the Result, then there's no way to determine whether it's a timeout or a normal return.

Either the iterator syntax needs to be changed to for item in iter, or try_fold needs to be used to integrate error handling, take_while, filter_map, and take. Either way, the changes will be more significant than they are now.

I see, I don't know that rust doesn’t let errors automatically propagate from a closure.