Feature: Proc-macro API

[dmgolembiowski]

Describe the solution you'd like

I'll need to explore the library to determine feasibility, but I'd like to decoratively build tasks within a Rust program, or manage particular profiles from a compiled program.

#[pueue_task(broker_name = "email_checker")]
async pub fn check_my_email(cnx: &Client, partial_fn: &PartialFn, callback: &Callback) -> Result<()> {
    email_checker
    .apply(partial_fn(cnx))
    .and_then(callback)?
} 

Naively, some partial function capability could be supplied by the library as a macro. Some intelligent person wrote this on crates.io or on the discord community server. (Credits to them)

macro_rules! partial {
    // The macro works with 3 lists
    // 1. closure args $cl_arg(s)
    //    The argument identifiers for the closure
    // 2. fn args      $fn_arg(s)
    //    The argument identifiers and forwarded expressions for the fn
    //
    //    Arg idents are passed around for hygiene reasons and to keep track
    //    of their number
    //
    // 3. the macro arguments $m_args
    //    A list of expressions and the forwarding sign '_'
    //    from which the former two lists are built up
    //
    // Until $m_args is empty, an element is popped off its front
    // and the appropiate pieces are pushed to cl_args and/or fn_args
    //
    // The fn ident and the move closure "boolean" (either "move" or "()")
    // are simpyl passed through during list processing inside $pt (pass-through)

    // exhausted macro arguments, create closure
    (@inner [(() $id:expr) ($($cl_arg:ident),*) ($($fn_arg:expr),*)] ()) => {
        |$($cl_arg),*| $id($($fn_arg),*)
    };
    // with move
    (@inner [(move $id:expr) ($($cl_arg:ident),*) ($($fn_arg:expr),*)] ()) => {
        move |$($cl_arg),*| $id($($fn_arg),*)
    };

    // process forwarder '_' ,
    (@inner [$pt:tt ($($cl_arg:ident),*) ($($fn_arg:expr),*)] (_ , $($m_arg:tt)*) ) => {
        partial!(
            @inner [$pt ($($cl_arg,)* a) ($($fn_arg,)* a)] ($($m_arg)*)
        )
    };
    // last forwarder (if no trailing comma)
    (@inner [$pt:tt ($($cl_arg:ident),*) ($($fn_arg:expr),*)] (_) ) => {
        partial!(
            @inner [$pt ($($cl_arg,)* a) ($($fn_arg,)* a)] ()
        )
    };

    // process given expr
    (@inner [$pt:tt $cl_args:tt ($($fn_arg:expr),*)] ($e:expr , $($m_arg:tt)*) ) => {
        partial!(
            @inner [$pt $cl_args ($($fn_arg,)* $e)] ($($m_arg)*)
        )
    };
    // last expr (if no trailing comma)
    (@inner [$pt:tt $cl_args:tt ($($fn_arg:expr),*)] ($e:expr) ) => {
        partial!(
            @inner [$pt $cl_args ($($fn_arg,)* $e)] ()
        )
    };

    // entry points
    // ordered to match eagerly
    // move
    (move $id:expr , $($args:tt)*) => {
        partial!(@inner [(move $id) () ()] ($($args)*))
    };
    (move $id:expr ; $($args:tt)*) => {
        partial!(@inner [(move $id) () ()] ($($args)*))
    };
    (move $id:expr => $($args:tt)*) => {
        partial!(@inner [(move $id) () ()] ($($args)*))
    };

    // no move
    ($id:expr , $($args:tt)*) => {
        partial!(@inner [(() $id) () ()] ($($args)*))
    };
    ($id:expr ; $($args:tt)*) => {
        partial!(@inner [(() $id) () ()] ($($args)*))
    };
    ($id:expr => $($args:tt)*) => {
        partial!(@inner [(() $id) () ()] ($($args)*))
    };
}

/// Possible Usage resembles:
fn main() {

    fn identity(x: u32) -> u32 { x }

    let mut n = 0;
    let mut f = partial!(identity => { n += 1; n});
    assert_eq!(f(), 1);
}

Describe alternatives you've considered

I have considered zero alternatives so far. :)

[Nukesor]

To be honest, I don't yet fully understand what you're trying to accomplish and how this would look like in a code base :smile:

You seem to have a very concise idea on how this would work, but I don't get the desired functionality from the #[pueue_task(broker_name = "email_checker")] snippet yet.

Could you further elaborate your feature proposal and maybe provide some concrete code examples :) This would definitely help me to better understand your request.

Also a fair warning when using the pueue_lib. I'm currently not 100% following semver :sweat: :

When hitting v1.0 I planned to keep strict backward compatibility. Not only for pueue itself, but also for pueue_lib. However, it turned out that keeping Rust API versions compatible in between each other is a huge development overhead and I just don't have the time and motivation to do so, as this is still a hobby project of mine. One alternative would have been to go for long release-cycles and regular major releases, which I'm not a big fan of.

For this reason, I had to compromise:

• All changes in schemas are usually documented in the CHANGELOG.md.
• Under some circumstances, versions might get incompatible in between each other, but I usually try to indicate this via a minor release.
• Schema changes are often introduced in releases, as the SemVer notation is mainly used to indicate compatibility to the Pueue daemon's API (which uses cbor to facilitate backward compatibility) -> The versioning does not always indicate stability on the Rust API level.

[Nukesor]

Another fair warning:

Pueue will keep its focus on being a neat convenience tool for the commandline. It can indeed be used as a programatic task scheduler via its APIs, but this will never be the main scope of this project.

-> If you want to add features that're only introduced to make Pueue a process scheduler backend, I might be hesitant to approve those changes ;D

That said, I'm curious what you're planning to do :)

[Nukesor]

Ping @dmgolembiowski

[dmgolembiowski]

Ping @dmgolembiowski

Oh yeah @Nukesor thanks for reminding me about this!

[dmgolembiowski]

Another fair warning:

Pueue will keep its focus on being a neat convenience tool for the commandline. It can indeed be used as a programatic task scheduler via its APIs, but this will never be the main scope of this project.

-> If you want to add features that're only introduced to make Pueue a process scheduler backend, I might be hesitant to approve those changes ;D

That said, I'm curious what you're planning to do :)

Disclaimer: I can't remember my inspiration for the broker_name or its intended significance, but I do recall wanting to emulate a kind of decorator pattern like the @celery.task from Flask. In theory, I think there's a lot of exciting potential for a magic procedural macro that moves IO and computation out to a worker pool member.

It might be interesting to demonstrate the possibilities with other crates, like xshell, telefork, ffmpeged, etc.

[Nukesor]

I see how a @celery.task style decorator could be nice for ergonomically spinning of tasks to a managed worker pool. Though, I have a hunch that doing this dynamically in Rust might turn out to be a bit tricky :D.

Still, I'm not sure how this would work in combination with Pueue, as Pueue is only capable of executing shell commands. It wouldn't be possible to just spin of any function to the Pueue daemon. Doing this in between processes might even be impossible in the first place.

[Nukesor]

I'm going to close this issue for now :) Feel free to re-open if you remember the use-case at hand :D

[dmgolembiowski]

@Nukesor While playing with the pueue repository and its assorted crates, I looked at what it might take to control the pueue daemon and pueue client with a more programmatic API -- for fun mostly.

Since a few essential things weren't public exports, I ended up duplicating a bunch of source code into pueue-daemon-lib and pueue-client-lib directories, dropping out the main.rs in the process. It feels amazing having the entire libraries right there, with handles to spawning clients, interacting with tasks, etc. and I think pueue makes for a wonderful inclusion in a ton of task-driven projects.

Plus -- the CLI experience it has is already awesome! It lends itself really well to old-fashioned system administration. But potentially having a batteries-included task broker with persistence after the program is a developer's dream. If it's alright with you, I think publishing a fully public client library -- with exposed handles to spawning the daemon, exposed control over the networking, etc. would be fantastic.

[dmgolembiowski]

Also, is public [a-t] ARN3.b33r a good way to reach you?

[Nukesor]

Plus -- the CLI experience it has is already awesome! It lends itself really well to old-fashioned system administration. But potentially having a batteries-included task broker with persistence after the program is a developer's dream. If it's alright with you, I think publishing a fully public client library -- with exposed handles to spawning the daemon, exposed control over the networking, etc. would be fantastic.

Publishing a separate crate that simply mirrors code doesn't seem like the best idea. Especially if it has the name of the original project :sweat_smile:. We would have to synchronize the publishing of our crates. Otherwise there might be incompatibilities for users when they upgrade Pueue and the external libraries haven't been updated yet.

I would very much prefer to either make the stuff you need public in the pueue_daemon_lib crate or move some client logic to pueue_lib.

Also, is public [a-t] ARN3.b33r a good way to reach you?

Yes you can reach me via that email or via the email I use to sign my commits :)