mutable filter state

[wseaton]

I have a use-case where I'd like to keep track of a limited amount of global state across invocations of the same filter, for example, keeping track of the number of bound parameters to a sql query. Currently we can't get a mutable handle to the env, only &Environment. Initially I thought I could do this by passing a reference to a HashMap or some other shared data structure around into the filter, but it seems like the filter functions need to remain pure due to the automatic type conversion that is happening (which is great, btw).

jinjasql, the library which I'm trying to emulate handles this via thread local variables, is there an idiomatic way to share filter state across invocations?

[wseaton]

I seem to have something workable using a combination of lazy_static! + Mutex<Vec<String>> to keep track of a global context, which allows the filter function to remain "pure" in it's inputs, but still do some book keeping.

Here's what this ends up looking like:

select * from
    (
    select * from {{ table_name | upper }}
    where x in {{ other_items | reverse | inclause }}
)a
where column in {{ items | inclause }}

ctx:

context!(
    table_name => "mytable",
    items => vec!["a", "b", "c"],
    other_items => vec!["d", "e", "f"]
)

Can render to:

select * from
    (
    select * from MYTABLE
    where x in ($1, $2, $3)
)a
where column in ($4, $5, $6)

--params: ["d", "e", "f", "c", "b", "a"]

What I'm worried about is the general thread-safety of this, along w/ any performance considerations and if there's a more idomatic way to do it w/ the current APIs.

[mitsuhiko]

So having a mutable environment won't be of much use, but I have been thinking of allowing extension types to be stored with the State object. So one could do something like this:

#[derive(Default)]
struct MyThing(Vec<u32>);

fn my_filter(state: &State) -> Result<u32, Error> {
    let my_thing: &mut MyThing = state.extensions_mut();
    my_thing.0.push(my_thing.0.len() as _);
    Ok(my_thing.iter().copied().sum::<u32>())
}

But that opens up some questions about how long this data lives etc. and so far I haven't found a lot of use cases for this yet.

[wseaton]

@mitsuhiko your suggestion looks great to me, I could then just create a QueryContext struct for book keeping the fields I need.

Right now I'm using

ref_thread_local! {
    static managed CONTEXT: Mutex<Vec<String>> = Mutex::new(Vec::new());
    static managed PARAM_COUNT: AtomicUsize = AtomicUsize::new(0);
}

Which seems to work but I have still have doubts is fully thread-safe :slightly_smiling_face:

[mitsuhiko]

I toyed around with it now and while it works there are some open questions with it. Namely how to initialize them. In a way it would be nice to be able to do this:

let mut extensions = Extensions::default();
extensions.insert(MyThing { ... });
tmpl.render_with_extensions(context, extensions).unwrap();

I just wonder if that doesn't make this into a very specific feature all the sudden.

[mitsuhiko]

I first want to collect some experiences with minjinja before committing to it. Thread locals are a pretty good workaround for now that lets one unblock. Since minijinja already requires thread locals anyways to work around serde limitations internally this is in fact already a pattern anyone needs to resort to anyways.

[wseaton]

@mitsuhiko sounds good. I'll keep iterating on my library w/ the thread local approach and follow along here for any API updates.

[mitsuhiko]

Since there hasn't been more demand for this I will close this as wontfix.