user state?

[mgreenbe]

Are there idioms/features for passing user state through combine parsers, like in Haskell's Parsec? (I guess, for labelling purposes, this could be considered a feature request.)

[Marwes]

There are basically two ways of doing this.

• Pass the state to the combinators directly

This is easy and fairly flexible but has the downside that the state needs to be shared between all the combinators that are used but if you can accept an immutable state or a RefCell (and perhaps a Rc) then this is the simplest way.

fn my_parser(my_state: MyState, parser: P) -> impl Parser<Input = I, Output = O> {
}

• Add a wrapper around the Stream type you use.

This lets you get a &mut to your state at the expense of slightly more complicated code.

struct StateStream<I> {
    state: MyState,
    input: I
}

fn my_parser<I>(parser: P) -> impl Parser<Input = StateStream<I>, Output = O> {
    // You can use the `function::parser` to get hold of the state again
    // We could add a combinator to make this easier as well
    parser(|input: &mut StateStream<I>| {
        // Do something with the state
        ...
        // Run the parser you want as normal
         some_parser.parse_stream(input)
    })
}

[Marwes]

https://github.com/Marwes/combine/tree/user_state / https://github.com/Marwes/combine/commit/48137f71c853f8607510dc568f6def32942599b4 contains an example of the second way. It may get merged into combine after some polish.

[michaeljones]

I am attempting to use the current combine::stream::state::Stream but I'm afraid I'm struggling. Mostly because I am still relatively new to Rust (though a few 10s or more of hours in.) I would greatly appreciate some example code to see how to use it.

Currently I would like to know how to access something like the State below in a parser like the type_name parser below. Could you help me understand how to do that?

struct State {
    pub indentation: i32,
}

pub fn type_name<Input>() -> impl Parser<Input, Output = String>
where
    Input: combine::Stream<Token = char>,
    Input::Error: ParseError<Input::Token, Input::Range, Input::Position>,
{
    upper()
        .and(many(alpha_num()))
        .map(|(first, mut rest): (char, Vec<char>)| {
            rest.insert(0, first);
            rest.into_iter().collect()
        })
}

[michaeljones]

I'm nervous to share my attempts as I don't like exposing my ignorance but I also want to look like I've tried so this was my very basic attempt:

struct State {
    pub indentation: i32,
}

pub fn type_name<Input>() -> impl Parser<combine::stream::state::Stream<Input, State>, Output = String>
where
    Input: combine::Stream<Token = char>,
    Input::Error: ParseError<Input::Token, Input::Range, Input::Position>,
{
    upper()
        .and(many(alpha_num()))
        .map(|(first, mut rest): (char, Vec<char>)| {
            rest.insert(0, first);
            rest.into_iter().collect()
        })
}

I realise that doesn't attempt to access the state but it doesn't compile. It errors with:

error[E0277]: the trait bound `impl combine::Parser<combine::stream::state::Stream<_, elm::parser::tokens::State>>: combine::Parser<Input>` is not satisfied
  --> src/elm/parser/tokens.rs:45:47
   |
45 | pub fn function_name_or_type_name<Input>() -> impl Parser<Input, Output = String>
   |                                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ the trait `combine::Parser<Input>` is not implemented for `impl combine::Parser<combine::stream::state::Stream<_, elm::parser::tokens::State>>`
...
50 |     function_name().or(type_name())
   |     ------------------------------- this returned value is of type `combine::parser::choice::Or<impl combine::Parser<combine::stream::state::Stream<_, elm::parser::tokens::State>>, impl combine::Parser<combine::stream::state::Stream<_, elm::parser::tokens::State>>>`
   |
   = note: required because of the requirements on the impl of `combine::Parser<Input>` for `combine::parser::choice::Or<impl combine::Parser<combine::stream::state::Stream<_, elm::parser::tokens::State>>, impl combine::Parser<combine::stream::state::Stream<_, elm::parser::tokens::State>>>`
   = note: the return type of a function must have a statically known size

Which if I knew Rust better might explain exactly what I need to know but I don't understand :/

[Marwes]

You are trying to use a parser that demands state::Stream where you only provide one on Input.

In the error

impl combine::Parser<combine::stream::state::Stream<_, elm::parser::tokens::State>>:
     combine::Parser<Input>

So your outer function_name_or_type_name parser must also demand state::Stream

As for accessing the parser state, you can use https://docs.rs/combine/4.2.1/combine/trait.Parser.html#method.map_input to access the input stream simultaneously as the output of a parser, https://docs.rs/combine/4.2.1/combine/fn.parser.html to get access to the stream directly allowing you to do whatever parsing you want with it or https://docs.rs/combine/4.2.1/combine/parser/combinator/fn.factory.html which is like parser but lets you return a parser instead of manually parsing.

[michaeljones]

Thank you so much for taking the time. I'm embarrassed. I got so lost in my early attempts that I didn't even notice that this error was for a different parser that was using the parser I've just changed. I am sorry for wasting your time. Thank you for the help and further guidance on accessing the state.

[michaeljones]

Definitely made some progress thanks to your help. I hope I don't mind me continuing to trouble you but at the moment the compiler is crashing :/ I'm getting:

error: internal compiler error: src/librustc_traits/normalize_erasing_regions.rs:35: could not fully normalize `<impl combine::Parser<combine::stream::state::Stream<Input, elm::parser::state::State>> as combine::Parser<combine::stream::state::Stream<Input, elm::parser::state::State>>>::PartialState`

thread 'rustc' panicked at 'Box<Any>', src/librustc_errors/lib.rs:875:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

note: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md#bug-reports

note: rustc 1.43.1 (8d69840ab 2020-05-04) running on x86_64-unknown-linux-gnu

note: compiler flags: -C debuginfo=2 -C incremental --crate-type lib

note: some of the compiler flags provided by cargo are hidden

error: aborting due to previous error

I think it is related to my attempts to use the parser! macros. I've got:

#[derive(Copy, Clone, Eq, PartialEq, Ord, PartialOrd, Debug)]
pub struct State {
    pub indentation: i32,
}

pub type StateStream<Input> = combine::stream::state::Stream<Input, State>;

in a state module and then:

use super::state::StateStream;

pub fn type_annotation_<Input>() -> impl Parser<StateStream<Input>, Output = TypeAnnotation>
where
    Input: combine::Stream<Token = char>,
    Input::Error: ParseError<Input::Token, Input::Range, Input::Position>,
{
    combine::choice((
        combine::attempt(function_type_annotation()),
        unit(),
        parens_type_annotation(),
        tupled_type_annotation(),
        typed_type_annotation(),
        generic_type_annotation(),
        // recordTypeAnnotation
    ))
}

// Wrapper for type_annotation to allow it to be called recursively
parser! {
    pub fn type_annotation[Input]()(StateStream<Input>) -> TypeAnnotation
    where [
        Input: combine::Stream<Token = char>,
        Input::Error: ParseError<Input::Token, Input::Range, Input::Position>
    ]
    {
        type_annotation_()
    }
}

In another module. And a further similar use below that:

pub fn non_function_type_annotation_<Input>(
) -> impl Parser<StateStream<Input>, Output = TypeAnnotation>
where
    Input: combine::Stream<Token = char>,
    Input::Error: ParseError<Input::Token, Input::Range, Input::Position>,
{
    combine::choice((
        unit(),
        parens_type_annotation(),
        tupled_type_annotation(),
        typed_type_annotation(),
        generic_type_annotation(),
        // recordTypeAnnotation
    ))
}

// Wrapper for type_annotation to allow it to be called recursively
parser! {
    pub fn non_function_type_annotation[Input]()(StateStream<Input>) -> TypeAnnotation
    where [
        Input: combine::Stream<Token = char>,
        Input::Error: ParseError<Input::Token, Input::Range, Input::Position>
    ]
    {
        non_function_type_annotation_()
    }
}

I can get the cargo expand helper to work and I can include the output from that if it would be useful but when I attempt to cargo build --lib it fails with the error at the top. If I comment out the only file with parser! usage in it then a number of other modules with other normal parsers in them seem to compile successfully.

I'm sorry if it is a basic mistake on my part. I kind of hope it is. I've not idea how to go about preparing an example for the rust compiler team though I guess as my project is open source I could just commit the current state. It fails with both 1.43.1 and the current nightly.

[Marwes]

As the error message says, it is a compiler error so not necessarily something you did wrong (at least I can't see anything obvious). You can try the nightly compiler to see if it has been fixed already, if not you can try searching for an issue on rust-lang/rust or post a new issue if it does not exist (ideally with a minimal reproduction but that might be difficult).

If parser! does not work you can try opaque! as an alternative to write recursive parsers.

[michaeljones]

Thanks for the quick response. I'll try to switch to opaque and see how that goes. I don't think I'm in the best position to create a reproduction for the compiler team but I'll see what I can find. Thanks!

[michaeljones]

I've just tried to switch over to using opaque instead.

With this code:

pub fn type_annotation<Input>(
) -> combine::parser::combinator::FnOpaque<StateStream<Input>, TypeAnnotation>
where
    Input: combine::Stream<Token = char>,
    Input::Error: ParseError<Input::Token, Input::Range, Input::Position>,
{
    opaque!(combine::parser::combinator::no_partial(combine::choice((
        combine::attempt(function_type_annotation()),
        unit(),
        parens_type_annotation(),
        tupled_type_annotation(),
        typed_type_annotation(),
        generic_type_annotation(),
        // recordTypeAnnotation
    ))))
}

And it also crashes the compiler. I'm just writing this in case others end up in a similar position for some reason.