Directory layer: Returns partial content for large directories

[pierrebelzile]

The code in node.rs "list_sub_folders" does not iterate through the range_options.next_range(). Therefore returning a list of subdirs only include whatever FDB chooses to return. It would be better to have pagination but that would require a change to the API.

The fix is trivial and I can create a patch:

pub(crate) async fn list_sub_folders(
    &self,
    trx: &Transaction,
) -> Result<Vec<String>, DirectoryError> {
    let mut results = vec![];

    let mut range_option_opt = Some(RangeOption {
        mode: crate::options::StreamingMode::WantAll,
        ..RangeOption::from(&self.subspace.subspace(&DEFAULT_SUB_DIRS))
    });

    while let Some(range_option) = range_option_opt.take() {
        let fdb_values = trx.get_range(&range_option, 0, false).await?;
        range_option_opt = range_option.next_range(&fdb_values);
        for fdb_value in fdb_values {
            let subspace = Subspace::from_bytes(fdb_value.key());
            // stripping from subspace
            let sub_directory: (i64, String) = self.subspace.unpack(subspace.bytes())?;
            results.push(sub_directory.1);
        }
    }
    Ok(results)
}

[PierreZ]

Good catch! You're correct, the current implementation doesn't iterate through paginated results from FDB.

Your fix works, but we should use get_ranges_keyvalues() instead of manually handling next_range().

Want to submit a PR? Please also check other get_range() calls in the directory layer for the same issue.