LendingIterator with Rust 1.65+

[fkohlgrueber]

If I understand correctly, the release of Rust 1.65 should make it possible to implement lending iterators without workarounds. The release notes even use lending iterators as an example of what's possible with GATs:

/// An `Iterator`-like trait that can borrow from `Self`
trait LendingIterator {
    type Item<'a> where Self: 'a;

    fn next<'a>(&'a mut self) -> Option<Self::Item<'a>>;
}

I've been using the code above in a library I'm writing and it's working fine for my use-cases. I don't like defining the trait in my lib though and would prefer using a crate that would also implement common iterator methods for the lending iterator trait.

My question now is whether this project will take advantage of GATs being stable now. For interoperability, it'd make sense to have a common library for lending iterators instead of each library defining their own version IMO. It's possibly even something worth including in the stdlib, but I didn't find discussions about it.

What do you think about this situation and what would you recommend?

[danielhenrymantilla]

Yeah, I intend to spend some time exploring usage of GATs over nougat in some cases (I might add a module for this, I think, with a LendingIteratorGat, for instance); but I haven't got the time for it as of yet.

Until then, I did post over Zulip a list of all the limitations Rust currently has for a clean (no workarounds) LendingIterator design:

• https://rust-lang.zulipchat.com/#narrow/stream/219381-t-libs/topic/lib.20after.20GATs/near/307867488

[Crazytieguy]

@danielhenrymantilla Hi 👋 I spent some time working on a version of lending iterators that uses gats, doesn't need the higher kinded types and have an over all cleaner API IMO. Instead of publishing it as a separate crate I wanted to check if you would want to modify this crate. Ideally I'd like to work together to create what will become the canonical lending iterator 😄

https://github.com/Crazytieguy/lending-iterator

[danielhenrymantilla]

So, I am open to adding a gat module for a GAT-based implementation, such as @Crazytieguy (feel free to open a PR doing that 🙏). I won't be replacing the main API until at the very least something like:

let index: usize = …;
some_lending_iterator.map(|x: &mut [u8]| &mut x[index])

is capable of working / resulting in another fully usable lending iterator.

AFAIK, right now the only way to work with .map() / .and_then and the like is by:

• either using free / non-closure fns for which we can write an explicit lifetime-generic signature;
• or to somehow annotate the closure itself with such a signature,
  • either on nightly through closure_binders:

    some_lending_iterator.map(for<'a> |x: &'a mut [u8]| -> &'a mut u8 { &mut x[index] })
    

  • or on stable through some helper macro which we could reëxport such as https://docs.rs/higher-order-closure:

    some_lending_iterator.map(hrtb!(|x: &mut [u8]| -> &mut u8 { &mut x[index] }))
    

In that regard, the current API of the crate using:

some_lending_iterator.map::<HKT!(&mut u8), _>(|[], x| &mut x[index])

or even the dedicated pervasive-case helper:

some_lending_iterator.map_mut(|x| &mut x[index])

still seem more convenient.

• (for those unconvinced about HKT! vs. hrtb!, the former works with inference, whereas the latter does not, so when u8 is, instead, some very complex type or a type which happens to mention generic parameters in scope, both hrtb! and closure_binders become quite painful to use; HKT!, on the other hand, lets you infer any part of the type that does not involve the higher order lifetime: HKT!(&mut _) would work, for instance (and .map_mut also Just Works™)).

[danielhenrymantilla]

For instance, something actually useful from having a gat module, on the other hand, would be for manual LendingIterator implementations:

struct MyLendingIter { … }

impl LendingIteratorGat for MyLendingIter {
    type GatItem<'next> = …;
    where
        Self : 'next,
    ;

    fn gat_next(this: &mut self) -> Option<Self::GatItem<'_>> {
        …
    }
}

and then magically having MyLendingIter : LendingIterator thanks to some blanket impl along the lines of:

#[gat]
impl<T : LendingIteratorGat> LendingIterator for T {
    type Item<'next> = T::GatItem<'next>;
    where
        Self : 'next,
    ;

    fn next(&mut self) -> Option<Self::Item<'_>> {
        T::gat_next(self)
    }
}

[Crazytieguy]

I see your point regarding the missing type inference, I agree HKT! is more powerful and we shouldn't settle for anything less powerful.

At this point, it's fairly obvious what the API will be in the long term, and it's possible to create a future-proof API today. I think having several "pervasive-case" helpers is fine, if we name them well we'll be able to deprecate them when map can handle all cases properly - and just not have map at all until then.

Gaining real users is valuable, and we can't gain many users before the API is simplified and future-proofed (people will want stability).

I can't see how adding a gat module will be future-proof.

Maybe consider it once more after all?

[Crazytieguy]

I don't think it needs to be a module if it's already a separate crate

I don't follow, is there a separate crate? I don't understand if you're suggesting creating another crate without HKTs or if you're suggesting something for this crate.

[WanderLanz]

Oh sorry I was referencing the example GAT implementation that you made. Not a lot of sleep when writing my comment, sorry it wasn't clear. Could definitely be a good module/feature of this crate if it ends up working well with generic implementations, but my gist was that it has plenty of merit as a standalone crate (a GAT lending-iterator interface) if not just to standardize a GAT implentation.