[Discussion] Moving Riker development forward

[leenozara]

Dear all,

I would like us to begin to discuss how we can move forward with Riker development, both from a code perspective but maybe also a wider project concept.

I created Riker a couple of years ago, which itself was a reiteration of a previous closed source project from the early days of Rust. Riker has come a long way and I’m pretty proud of the result. I must say that the best updates to the project have come from others discussing and making PRs, rather than myself. Thank you to everyone that has given their time to improving Riker!

For Riker to now evolve, for it to see its potential, we must change the way the project is developed. It’s simply not enough to have one person setting the direction, without communication, and then relying on others to pick through code to see where changes might be useful. Collaboration needs to start at the design and roadmap phases. We need thoughtful discussion, a sense of collective ownership as well as specialist ownership of specific systems.

To be clear, I am committed to Riker. This isn’t me offloading the project to any group of people that want to take it. My skills as a Rust developer however aren’t nearly as good as many of you and my value is more on actor system design and concepts. I’m also in a position to fund server/hosting/build costs that save time, etc. If we continue to evolve I imagine other costs will emerge.

We’ve always tried to maintain a few core principles:

• Riker isn’t reproducing or competing with Rust’s concurrency, i.e. Riker actors are not a unit of concurrency like they are in Erlang.
• Riker isn’t a complete stack including HTTP servers, e.g. like Akka is for Scala.
• Riker makes it easy to work with state concurrently, including changes in state and behavior.
• Riker makes it easy to build complex systems by using actors to represent parts of the system and providing guarantees.
• Provide excellent documentation (or at least try to).

I had originally planned to create a roadmap, contribution guide, GitHub labels, etc and communicating this to everyone earlier last month. However, it became clear that even these must be done collaboratively.

So with a bit of the background laid out, what I would like to do is invite you all to share your thoughts and ideas here. The immediate main two questions I have are:

1. Is there interest in forming a collective development of Riker from you?
2. If so, what does that look like? (What are the best practices? How do we organize?)

Let’s keep in mind that it might take some contributors a few days to see this issue. I’d like to give enough time to hear from new and old members of the community, as well as any one who would like to join but hasn’t yet.

Finally, this is certainly a discussion. Not limited to sharing ideas specifically with me, but to actually discussing between us all.

Thank you all and I hope everyone is starting safe. Looking forward!

[igalic]

We’ve always tried to maintain a few core principles:

• Riker isn’t reproducing or competing with Rust’s concurrency, i.e. Riker actors are not a unit of concurrency like they are in Erlang.
• Riker isn’t a complete stack including HTTP servers, e.g. like Akka is for Scala.
• Riker makes it easy to work with state concurrently, including changes in state and behavior.
• Riker makes it easy to build complex systems by using actors to represent parts of the system and providing guarantees.
• Provide excellent documentation (or at least try to).

i think this is excellent to have. i think all projects should be upfront and obvious about it, so that people who want to use this project, or extend it know when they're in the right place.

The immediate main two questions I have are:

1. Is there interest in forming a collective development of Riker from you?
2. If so, what does that look like? (What are the best practices? How do we organize?)

i have previously co-founded, and helped build a volunteers-only community, so i have ideas and opinions on how to do that, at least at a certain scale https://voxpupuli.org/blog/2018/02/06/voxpupuli-community-journey/

[aav]

the core principles sound good to me. I can commit a certain time to the project since I believe that it is crucial to have an actor component, to build anything serious. My experience with Rust is not so big - ~1 year. But I have a very long history with Erlang, which certainly may help.

[hardliner66]

I'm definitely interested in developing riker.

The core principles of riker are nice, but they should be written down on the web page and on github. Right now it could be interpreted to be more than "just" an actor framework. It should be clear what you get as a user of riker. And it probably would be better to remove features from the webpage if they are currently not implemented and moved onto a roadmap.

A main point in the mid- to longterm should be forming a core-team, so that reviewing and merging of pull requests doesn't rely on only one person.

Best practices

• A lot of projects seem to adopt some kind of RFC workflow to make proposals and define how they should work. I don't know if this is necessary at the moment, but it provides a defined way for people to contribute new ideas and features.

• Another thing which I have good experiences with, are workgroups. Something like a performance workgroup, which focuses on optimzations or a documentation workgroup making all the documentation for riker. This way issues can be assigned to a certain workgroup which can independently work through them. Workgroups can also have their own meetings/discussions so that the main meetings don't drown in issues which are too specific for a generic meeting.

• As I just mentioned, there probably should be some sort of meeting (Discord, Skype, etc.) some time during the development cycle, like at the end/beginning of each version, in order to coordinate what should get done next.

Finally, some things I would like to work on or see in riker some day

• Backpressure: We already talked about this on gitter. I definitely would like to see some solution for backpressure in riker. I don't know if I have enough knowledge to help in this area, but I think this is something which should definitly be handled somehow.

• Performance This is a topic which I'm currently learning about because I need it for my day job. And I want to apply what I learn to riker. I already made a commit to lower allocations in riker and I would like to do more in this area including benchmarks etc.

• Architecture & Tooling I don't know if the architecture of riker currently allows something like this, but it would be nice if we could implement some dev/admin tools for riker like the Orleans Dashboard. I think the architecture should allow extending riker from the outside. Something like middleware in http-frameworks.

[aav]

Backpressure is one (but not the only one) way to handle the overload. IMHO, on the lowest actor level, it's better to have the most primitive 'fire and forget' send semantics, which is completely non-blocking. As a safety measure, it should be possible to limit actor's inbox size, and simply crash the actor in case the inbox reaches some size.

This is sufficient to act as a basis for many different load handling approaches, i.e. backpressure, load shedding, all sorts of pools, and load distribution tech.

I understand that this is the "Erlangish" approach and I'm not 100% sure if it will be idiomatic for Rust async world. But I think, it's worth discussing.

[leenozara]

I agree that forming a core team is what we need to as soon as possible. I also agree that an RFC process and the implementation of workgroups is the direction to head.

A few questions and ideas:

• Core Team to consist of two or three persons, for now
• How do we select for Core Team? Based on familiarity of Riker and Rust?
• RFC process is good, even now, since it also attracts other developers
• When is an RFC encouraged/required?
• Contribution guide to be created

An important point was raised by @hardliner66 regarding what is Riker and how this should be communicated. The original vision was to replicate Typesafe/Lightbend’s Akka framework, where it made sense for Rust. Akka isn’t just an actor framework but a framework to easily build scalable projects on actors, with additional tooling for testing, monitoring, etc. Akka provides a path for users from small apps up to large multi data center deployments. Much of this is achieved not by Akka specific development but by making it easy to bring existing solutions together.

I continue to see this is where a large amount of potential exists for Riker, since unlike JVM and Erlang, Rust can also be used (as a primary language) in resource limited environments such robotics, machines, SBCs, browsers. This is a natural area for actors to be applied, e.g. building a drone with all its systems and inputs represented as actors makes sense. This is all in addition to the existing potential in cloud-based applications for social, FinTech, Crypto, etc.

I’d like to hear from others about how Riker is solving problems today, or how they would imagine it might in the near future. The website and documentation should make a clear case for the project and what problems it aims to solve.

Thanks for the input so far.

[hardliner66]

@aav I'm not quite sure if a simple fire and forget is the best way to go. Especially the part about crashing an actor when it's queue gets full seems wrong. But I don't think this is the place to discuss the details. That should be explored as a workgroup and be backed up with tests or prototypes.

@leenozara I agree that a smaller core team would be good to get things going. Because I think there will be less friction if less people are involved. Once alle the core features are in place, it might be better to expand the core team.

I think a core member should know rust, know the riker codebase and should identify with the core principles. Otherwise it would be hard to work in the same direction. But I don't know the best way to select them. At last it's up to you to decide, which people would be most qualified, since it's your product and you should know best what you want riker to be and who can help you acheive that.

I also think that riker with rust has great potential. Compilation to webassembly is one of the things I'm very excited about. To compile on embedded platforms, no_std support would probably be needed, which would mean to get rid of most heap allocations. This should also be explored and tested.

[aav]

@hardliner66 perhaps, it's worth creating a separate issue to discuss backpressure, etc.

@leenozara at the end, you as the project creator, decide on how to steer the project. What is very important is to define core principles and project scope. Simply speaking, what Riker is and what it is not. Otherwise, it's possible to get into a very broad discussion.

[aav]

I’d like to hear from others about how Riker is solving problems today, or how they would imagine it might in the near future.

As mentioned above, I'm still in the exploring/learning phase. As a learning example, I'm working on a small home automation server, that interacts with a relatively exotic Zigbee controller. Currently, it is based on plain rust async framework. For me, it is obvious, that it would be very helpful to have an actor runtime as a basis for such software.

My dream would be to have a layered actor runtime

Layer I

• hierarchical actor tree, with built-in linking and monitoring (parent is informed on child termination, children get terminated on parent termination).
• clearly defined and document semantics and guarantees
• local actor registry

Layer II

• supervisor actors, with the ability to plug in their own supervision strategies.
• server actors supporting tell/ask interactions

Layer III

• advanced local registry (multiple registrations, property-based registration)
• broadcasting facilities (pub/sub channels, events, etc)
• queue management primitives
• monitoring and tracing

Layer IV

• distribution with pluggable transport protocols
• distributed registry
• All the rest :)

I clearly understand that this does not fully correspond to the current state of Riker, but I still wanted to write that.

[leenozara]

@aav I think you just wrote our roadmap :). But in all seriousness, I agree with this largely and has been how I’ve envisaged Riker from the beginning, at least the core. I appreciate how you’ve laid it out in terms of layers. We already have layers 1 and 2, mostly. This ties in nicely with the time I’m taking to reorganize the project - so I appreciate you sharing this.

[leenozara]

Just to update everyone briefly. The input provided so far from @igalic @hardliner66 and @aav has been invaluable. This discussion will continue to remain open for some time, so do voice your ideas if you have them and have not done so yet.

I’ll be sharing specific proposals soon regarding the core team, RFC and near-term community setup objectives.

Something that came up a couple of times was the idea that this is my project or product. My view is this: We are in the process of creating an environment that has structure and processes to make development of Riker manageable (and more fun). So let’s just go one step further and make the project owned by a group that share the same vision. That is the direction I hope to take. It can’t be done immediately because the project isn’t large enough to handle disparate views from just a handful of people. But I am looking at traditional ‘foundation’ like setups and also more modern DAOs. The key here is not to overburden the small group we are now, while at the same time setting a course for growth that includes group ownership of Riker.

[olanod]

Just wanted to sneak in to say hi and that is nice Ricker will keep going, the idea behind Ricker is very appealing and seeing people's commitment on moving the project forward is what I needed to start learning it. I'm no expert in the actor pattern, no Rust expert yet either so there's not much to contribute yet but hopefully some time down the line I can be of more help to this community. :)

[leenozara]

Hi everyone, I've created a repository with an initial document PR that outlines the Riker reorganization. There's some background on the project, how I see the opportunity and how we could start to organize around a common vision.

As always, I greatly value your input. I've also reached out to some on Gitter DM.

https://github.com/riker-rs/organization/pull/1

[shafqatevo]

Hi, what's the status of this project now - is it stalled for some reason? It'll be great to get some updates from the maintainers.

[hardliner66]

The project is currently in hiatus. There are some pull requests/issues coming in, but we have no one with write access. The person with write access hasn't answered in over a year by this point. They have probably a very good reason, personal life is more important than some project on the internet. So until he comes back, the only way forward would be fork and rebrand, because the riker domain and riker-rs organization on github are already used for this project.

You might be asking:

• why not wait? This whole thing already happened once. Last time I was still in the "wait for a return" camp. Now it happened again and no one has the access rights to take over.

• why not fork? If someone thinks the way riker works internally is a good enough reason to go through the hassle of rebranding and maintaining a fork, please do. More projects for people to learn from is always a good idea. I have looked into other actor systems and think there are more mature projects to put energy into. I'll be looking into bastion for instance, because it already has stuff like supervision and has some nice concepts under the hood.

All in all, I still think that riker in it's current state is usable for simple use cases. So if you only want to experiment with an actor system or you have a simple use case, you can probably use it without problems. If you have something more complex or hit an edge case, you're probably on your own. I will still check the repo sometimes to answer some questions, but I wont be actively working on riker anymore (reviewing pulls, fixing bugs, etc.).

[shafqatevo]

Thanks for the information @hardliner66.

[leenozara]

@hardliner66 @shafqatevo and all,

I wish I could be active in Riker project management and development. Maybe I can come back to the project in the future but there's no guarantees. I tried to establish shared ownership and responsibility with the community but it seems I wasn't able to make a compelling enough reason to make it happen. I've resisted doing quick PR reviews just to get code merged because I would prefer to have Riker remain committed to the original design principles, something between Akka and Erlang actors.

At this time, if I were to commit myself to giving Riker the time, the love it deserves it would impact other businesses, the work and jobs of other people. As much as I'd like to, I can't alone manage Riker for now. I will say though that I believe more than ever that an actor system development framework for Rust is ideal. We're seeing continued growth of Rust - it's expanding to be applied across many use cases, moving further away from the metal, on to blockchains (Solana) and more. As robotics becomes less about the hardware and more about the software and systems, the more a framework like Riker is needed.

If anyone has ideas about how to move the project forward, github, website, etc., I'd be more than happy to hear.

@hardliner66 has mentioned another actor system project for Rust called Bastion. I've looked at this and can happily say that the fundamental design of this appears to be sound. If you're looking for an actor system you might want to see if Bastion is more actively developed and explore that project for your needs.

[hardliner66]

@leenozara first of all, thanks for the response.

I get the reasoning behind the intent to stick to the original design goals, but I don’t necessarily agree with that being a good thing. There are certain things that are just different in rust than in Java or erlang, which should influence stuff like api design. Furthermore, I think that what makes erlang/akka so good is more than just how the api looks. It’s about the runtime, stuff like supervision, etc. Also, many merge requests just fix things that are broken, without changing the design, so using it as an argument is disingenuous at least. Also, there were people willing to help out maintaining riker as well. So with a bit of communication it shouldn’t have been too hard to find some sort of solution, even if just temporary until a proper governance system would have been in place.

A way forward: IMO: it’s fine if you don’t have the time to review and merge every change yourself. You can let others do the review and give you a short version of the changes. Or hold a meeting every other week where people can explain current merge requests for you to give your okay. This way things can move forward and you can still keep an eye on what changes.

The only thing that doesn’t work at all, it’s going completely silent, not even responding to DMs. It’s fine if you have stuff to do, you can just say so. Just communicating would have helped.

I still think riker can have a bright future, but to truly cover the use case you talk about, we would need at least no_std support, supervision, some kind of introspection and persistence working. But all of those are stagnant. Also, none of these require changing the design too majorly, even tho it might be a good idea to evaluate that as well.

TLDR: If you don’t want to give away merge rights, do somewhat regular meetings. Make them as short as comfortable for you and just do one merge request after the other.

I would also recommend to talk with @milgner and/or their company if they want to take over the project, as they might have the resources to manage the project.