discussion
discussion copied to clipboard
rust-ml
Group organization and logistics
Hello!
Per some discussion in #1 What do we want to build?, I'm opening this thread as a place to start a discussion about what a Rust ML working group (which, at least for the moment, is unofficial) could look like, and as a place to coordinate tasks and structures, etc.
Based on some of the other WG structures, I think there are a few things in particular that might be worth some discussion. After there's some consensus built around those (and any other topics that might come up), either opening a PR with the new information on this repository's README or building a new rust-ml / wg coordination repository with that information might be in order. If there is a topic missed below that you would like to discuss, feel free to chime in! With any luck, once things here are worked out a little more, we can start breaking things down and begin digging into designing and writing code.
a) What is the initial plan for scope of work, and how can we break this down into smaller projects that can make decisions and coordinate within themselves? Machine learning is a really broad domain, so a well-stated initial scope of work and direction seems important. Scope creep or trying to start with an overwhelming number of goals rarely lead to success. There was some discussion in the aforementioned thread about putting together an outline for a Book, and then developing the ecosystem to match the work required to finish that.
Not to get too far ahead of ourselves in terms of breaking down a new group into even smaller groups, but considering how broad the field of ML is, having an idea of how to structure the different parts of that in a reasonable way seems important. Groups like Embedded seems to break things down into formal project teams, while others like Async and Secure Code seem to have kept things a little more informal in terms of structure. Personally, I think based on the breadth of the ML domain and scale we're currently at, a little bit of both is probably in order.
b) How can new contributors get involved? Based on the discussion thread, it seems like there's a decent amount of community interest, so making it clear their contributions are be appreciated and providing a clear path for providing them seems like it would be valuable. This might include some guidance on how marking issues with certain flags such that new contributors can easily find them and guidance for what's expected in terms of submission content.
c) In what way and how often should the group expect to meet? Across existing WGs, it seems like there's a tendency towards weekly or biweekly meetings on Matrix, Zulip, or Discord servers at a specified time. Outside of that, GH issues like these seems standard for tracking discussions. I've personally found a quick progress update or summary of work posted at regular intervals (perhaps after meetings) to be really helpful in keeping track of things. That's a task I'd be willing to volunteer for, if others think it might be helpful.
d) What pre-existing components should be used, and what do we think should be built from scratch? There's a fairly large amount of early-stage or abandoned Rust ML crates and projects. The ecosystem has seemed to center around ndarray, but there are certainly alternatives for other things where instead of building something from scratch, an abandoned or external project could be forked, or included by building a shim layer/interop functions. On the other hand, some other projects (particularly ones with poor documentation or strange design choices) might not be worth the time and effort spent trying to grok them and building things from the ground up might make more sense. Many Rust projects do a really good job at providing well thought-out interfaces for both people and code, and I know that's something that I'd like to see in any Rust ML project as well.
I haven't been very active recently, but I am still very much following this discussion - I am happy to contribute linfa to a working group organisation if it does indeed formally start and there is willingness to move the project forward. As I stated from the beginning, an ML ecosystem cannot happen unless a community organises to make it happen - solo efforts won't cut it.
Completely agree, I’m a bit busy today, but I’ll edit this or comment again later with thoughts and a path forward to organize the group and resources to start setting up.
Edit: unfortunately having a hard time focusing on anything right now :/
Stay safe everyone I’ll post my thoughts soon
I don't want to dominate the conversation, and am definitely interested in hearing other people's thoughts (especially contrasting opinions or things I didn't consider!), but I've had some spare time recently and I figured I'd try to get the ball rolling by putting some of my own thoughts out there.
a) I don't really want to pull in the entire discussion from the "What do we want to build?" thread, but to provide context, my personal end goal (at least for now) is developing a Rust-based soft real-time object detection program (ex. YOLOv3) that can be deployed to an embedded Linux SBC, ideally with something like Darknet with easy tooling to train on custom objects, but in Rust so I don't have to worry as much about shooting myself in the foot when developing the parts around it or dealing with complicated deployment to different targets. I'd also like to have that build process be easily replicable and modified by other people. Obviously, other people will have different goals, so breaking things down into smaller bits that can be used in other ways matters!
There's a lot of work between here and there, but I'd say that building an multi-class image classifier using using multi-threaded CPU computation as the primary backend, with documentation explaining how that system is implementing a neural network, and code templates for easily adding custom classes would add a lot of value and require pieces that could be re-used by people/groups with different goals.
That would mean:
- Reading images and converting them into usable data structures (
ndarray-visionis already working on this, I believe. I've also done some with integratingndarraywith theimagecrate andopencv-rust) - Writing a library for building a neural net with more than one layer, with at least a couple types of layers and activation functions available
- Developing tooling around directory structures for training/validation examples, templates for adding classes, and doing so in such a way that it can be used in production workflows to add new data on the fly
- Saving weights and easily reading past ones, such that classifier doesn't need to be co-located with the training environment
- Documentation explaining all of the above
As mentioned, I think the first one is already ongoing, and is an avenue for contributions. The neural network library would likely be its own objective. Defining the interfaces through which a neural net library accepts and writes data could be done early on, at which point the reading data and writing weights in a way that can be interchanged with other ecosystems could be worked on independently. I would imagine that the tooling around training and validation would naturally arise in the process of building such a library as the developers found itches they wanted to scratch, although my own experience in building true production workflows of this kind is limited, so I might be missing something there.
b) In retrospect, I should have put this question below c), because I think the answer is encouraging interested parties to pop into the Zulip/Discord/Matrix server to say hello. Also, encouraging the use of issue labels like help wanted or good first issue seems like a good pratice to encourage. Once the bones of any initial scope of work is put into place, I think making sure that the documentation is well-written and accessible to new contributors would have a positive impact as well.
c) I'd like to create a poll in the "What do we want to build?" topic to see what those who have already commented would be interested in doing, with the options of Matrix, Discord, or Zulip, with a synchronous meeting weekly or once every two weeks.
For whatever it's worth, I like Zulip the most between those options. It has support for multiple discussion streams, which I think would come in handy for a group that might be juggling multiple objectives at any given time, as compared to a single room at a time with Matrix. It also seems to be the most-used solution among active Rust working groups, with GameDev using Discord (makes sense, considering), and Embedded using Matrix. I'm also partial to meeting once a week, at least to start, and depending on
I'd also be happy to put together a small blog post somewhere (a Rust-ML Github pages site, perhaps?) every month or so with updates. I'm still a relative beginner in the ML field, and I think it would be a good way to learn while also making some sort of helpful contribution.
d) I think there's some consensus around this, but I do think that using ndarray as the core linear algebra library for a machine learning project is the right move. I've used it myself for some toy neural nets, and found it wonderful. The documentation's also good, with even more on the way, and there's a good number of projects floating around that use it as their own base already. I put together a write-up of my thoughts about using the rusty-machine, leaf, or juice repositories as a basis for a machine learning project, but the summary of it is that while I like the a lot of the design patterns for non-NN-related work in rusty-machine (which seems similar to the kind currently used in linfa, as mentioned) and really like the NN ergonomics of leaf/juice, I'm not really supportive of using a fork of any of the three as a basis of a general ML project, and would rather see something built from the ground up. I do like the idea of pulling in linfa (either for only pre-processing, or as the starting point of a full machine learning project in earnest) and existing code that do things that aren't in the core ndarray repository, like using ndarray-vision or ndarray-csv for getting accessing external data resources.
First thank you so much @quietlychris for all of this, great work! Also thanks very much to @LukeMathWalker er for the first sacrifice to the WG absorption monster (linfa) and the space to talk about all of this!
Structure
Yes I agree ML is very broad, we definitely want to start with the building blocks of ML (e.g. a lot of the linear algebra libraries, and the data formatting). So think at first we will have a pretty straight forward goal, get a model producing ecosystem that can do the basics out, and focus on the ergonomics of doing so. As we mature I see this getting much more convoluted and chaotic!
Contributing
I would definitely like to see Issues and PRs for everything we do, so seeing also flag/tags for everything we do would naturally flow into that. We should also focus on only one chat medium. Fracturing could be really painful for new users, especially on a new WG.
Meeting
I agree, having a “hot wash” or recap of the meeting in some word from by a minute taker (could be the same person or different for meetings) would be really helpful. Keep that to bullets (and things we can take action on, possibly even naming the said person to do it) are all we need here I think.
As for a schedule I think a lot of groups meet monthly when there’s not much to do or ramp up as there is a lot of work to once every two weeks, or once a week. At least that is my experience. Seeing as we are all from very different time zones we will need to find a good time that works for a majority of the group and make sure our text or video is recorded as best as possible so people can participate async.
As for the mediums, I think Zoom (you can record video and text chat, and it seems to work well with many people) for video meetings if we need, and Zulip for a normal persistent chat. Again those are my personal preferences they aren’t strong but I think they work well.
Current Environment
I think this is the first major task for the group, explore the current environment with some effort, and take tasks away to contact the authors of promising crates for first involvement in the working group, and making sure they are open to new contributors from the WG community. If they have abandoned the project it will be really important to ask permission to absorb the crate into the WG GitHub group, of course always attributing the original author(s). However, this will give the WG the ability to keep it up-to-date with dependencies, bug fixes, or discovered vulnerabilities.
Future Environment
We should also strive to write a manifesto-like document in our repository that states a bit about what we envision machine learning to be like sticking to the rust language. I would like to see a “In the future you can download yolo-rust from <FOO> and use the weights trained from <BAR> with a few lines of Rust. Etc etc, I’m still thinking what this should look like but that’s what I have so far
Membership Roles and Limits
We should also think about how big we want to group to be, I’m so excited that a lot of people are interested in bringing machine learning (or data science) or Rust as a first class tool, but we also need to acknowledge that “too many cooks in the kitchen” exists and we cannot grow too large too fast, I am not up for herding more cats online.
As for roles, we might want to designate classes such as ML-researcher, developer, Ergonomics Expert, etc. This way members can know who to lean on for what (for example I’m a developer working in a shop that does a lot of machine learning please don’t ask me about the latest white papers I won’t know!). Solidifying these roles should be a first tasking as well.
—
As for getting the ball rolling I think we might be getting ahead of ourselves with definitions work too specifically. First we need to gather a good core for the WG and definite what we like to see and how it will benefit Rust as a whole. This should keep the focus tight and best serve the community.
So for that I would like to see some consensus on:
- Founding Members
- Meeting types and schedules
- Asynchronous chat platform
- Meeting planning/minute storage/access for everyone
- Calendar for the WG
- Member roles/assignments
- First meeting with introductions and agenda for any decisions we haven’t made
After we get the group setup and organized into a repository with some simple action items out of the way (website, documentation on the group, joining/contributing, etc) then we should look at what we want to build.
Honestly I might even say we should look at the environment first before talk about recreating DarkNet or even ImageNet classifier to get crates up-to-date and selected the ones we want to pull into the WG to maintain or start to build because they simply don’t exist right now or are so old/in such a bad shape that it would be easier to just start over.
I like where you are going @quietlychris and I don’t disagree but I really want to crawl and walk before we run. Does that make sense?
I definitely understand your concerns regarding trying to move forward without a solid foundations, and in many ways share them. That said, I know that I personally find it difficult to work in a vacuum (not literally, space stuff is actually pretty fun). Extending your analogy, my intention wasn't to suggest trying to run before crawling, but I find the context of "hey, we're trying to move from this spot here to that spot over there", regardless of speed, to be helpful. Apologies if that's not what was actually communicated! I think that your thoughts in "Future Environment" would satisfy that.
I believe you and I are on the same page regarding the potential scaling issues of a group like this, as well as the difficulty coordinating synchronous communication in a volunteer, distributed environment. In any case, Zoom meetings have worked well for me, although depending on the number of attendees, have the potential to get a little frantic.
I agree it would be good to put together a list of promising crates to consider contributing/reaching out to. Your comments about taking care in the contribution/absorption process are well said.
Even though we're not a formal WG at the moment, based on similar group's pages (small overview here) I think it would be good practice to have a front page with the following information: mission statement, goals (current, long-term), how you can contribute, and meeting info, plus any necessary licensing info.
My thoughts on some of your bullet points:
Founding members: Hopefully I'm not being too presumptive by putting together this list, but here's a list of people who have repeatedly and recently expressed interest in a group like this: @deg4uss3r, @LukeMathWalker, @jbowles, @tiberiusferreira, @xd009642, and myself. I'd like to check in with them to see if that is still the case, and of course anyone I may have missed.
Meeting types and schedules: As mentioned, I'd be open to either chat or Zoom meetings. If we can get buy-in from at least a couple more people, an initial Zoom meeting at some point in the next week or two might be the way to go, and then we can hammer out some plans for an asynchronous platform, future meetings, and next there.
Asynchronous chat platform: As above, tentatively leaning towards Zulip at the moment
Meeting planning/minute storage/access for everyone: With the obvious caveats, I'd like to propose that Founding Members are given access to the rust-ml repo, with a new rust-ml / wg repository created. We can add a directory for meeting minutes (including meeting notes and action items moving forward) with a new dated file added after each meeting, and meeting planning can be a new issue that is created/closed once per cycle.
WG Calendar: I think the actual role of a calendar could be filled by the meeting planning issue and meeting minutes. It's possible I'm missing a good reason to have an actual separate calendar, though.
Member roles/assignments:
I agree with your concern regarding too many cooks in the kitchen, especially while we're trying to build a coordination structure. I've also seen groups with some top-level similarity to this experience disruptive amounts of contributor churn, which is also something to keep an eye on.
I like the idea of general assignments in terms of providing everyone in the group with an understanding of what the others are working on. That said, I'm not sure that formal roles or classes are necessary unless/until this group becomes much larger. After an initial introduction, I think those roles will probably arise organically. Instead, I think that there should just be some stress on not working on things that people are not qualified for, or at least being open about that. For example, I have no experience with SIMD code; therefore, I won't contribute to any SIMD code unless somebody more experienced is also working on it and I'll make sure those people review my code and are aware I'm a relative novice. I'm hopeful people will gravitate towards and take some ownership of the areas that they're most qualified for and interested in, which will help with that.
If we're interested in something like having classes, even having an ongoing bio of contributors and their areas of interest and expertise might be the way to go, rather than trying to abstract that information into named roles. I'm interested to what other people think on the subject.
First meeting: As suggested above, I'd like put together a Zoom/video meeting at some point over the next week or two. My own schedule's extremely flexible at the moment, so I'm interested in hearing when other people would be available. Having introductions and touching base regarding all the things mentioned in the above post seems likely agenda-wise, with the goal of establishing a list of concrete action items for moving forward.
Sounds good, I'd be interested in contributing. For chat platform I've not used zulip I tend towards discord but I'm generally easy
@quietlychris we are definitely on the same page, sorry if I came off a bit harsh I didn't mean to at all. I just wanted to focus on the logistics and group members first and try to keep the discussion in separate places so for new people they don't have to read everything at once!
@deg4uss3r No worries, I didn't think you did! I just realized my comments may have been a bit unclear in a couple places and wanted to try to address that. Hopefully, we'll hear back from a couple more people over the next week or so and can move forward from there.
Nice to see the organisation taking shape.
Answering @quietlychris, yes I'm still very much interested in this group. I currently also have flexible schedule and don't really have a chat platform preference. I already use Discord because the Rust Discord Server is very active and advertised on the front page of https://www.rust-lang.org/community, but I don't mind using another platform.
As a side note, I've seen ndarray mentioned here as a basis on which to build the ML foundations since it already has quite an ecosystem of its own. My concern here is the lack of GPU/Accelerator support (for deep learning) and the possibility that we might force ourselves into a corner there.
Maybe we could define Tensor operations as a Trait and implement them for different backends. I tried it when doing some experiments because I wanted to have both and Ndarray backend and Arrayfire one. Maybe this is not the best solution, I just wanted to raise this concern. We can discuss it further once we have a chat platform.
@tiberiusferreira Awesome! I definitely agree about the lack of GPU and support for Deep Learning, that's one of the things I hope we can add as a community rather than force ourselves into a corner.
I think that's indeed possible, we will just need to take a while as a group and do some requirements gathering first and try to prioritize!
Initially if we pick and create a server on the given chat platform can start doing things like adding a channel for existing crates in the ecosystem and building a view of what's available, what people are already working on and get an idea of who's around and interested then move onto having proper meetings and organising better.
@xd009642 That makes a lot of sense to me! I somehow missed this entire rust-lang community on Discord, which seems to be home to chat for a lot of different groups. Now that I know, I'm sure that I'll be keeping an eye on things much more there, and therefore think it might make more sense to go with a Discord server than not (all the same advantages as Zulip, save for not being open-source, but hey).
By my count, for people that have been mentioned/participated in this issue, there's 3 people with general preference towards a Discord server, one with a general preference towards Zulip, and two who have not expressed a preference. Since I think I'd like to start getting things moving, unless somebody expresses a strongly different opinion before tomorrow morning (4/1/2020) or otherwise disagrees, I'll create a new Discord server for this group then and post the details for joining here.
Also, @tiberiusferreira, I think your concern is valid, and is something that can and should be discussed once the org situation is a little more stable. In the meantime, I'm not sure if you've seen it (it didn't appear in your write-up, although that may be because it's currently inactive), but the gpuarray-rs crate addresses some of those concerns with integrated OpenCL for it's parent project deeplearn-rs, although I don't believe that it is fully featured GPGPU. If you haven't, it might be worth reading through their code to see if the methodology could work for other things.
@quietlychris I agree with you. Thanks for pointing me to gpuarray-rs ! I've also recently found out that emu has migrated to using WebGPU, which is quite exciting!
@quietlychris while my preference isn’t for discord, I wanted to make sure the terminology was clear!
We wouldn’t create a whole server but rather a channel under “Domain Working Groups” on the official server. I think @XAMPPRocky is the best person to help with that (sorry for the pull into the discussion if not)?
It would also be very helpful (at least for the GPU side) if we had someone who was in or participated in both the GameDev WG and this one.
@deg4uss3r I was operating under the assumption that since we (as of yet) have not been assigned an official status as a language WG, starting our own discrete server would be the path forward (either on Zulip or Discord). If we were to be allowed to operate on either of those, then I'd fully support creating a channel or stream for this group under those umbrellas, depending on XR's response to your comment.
Hopefully not to needlessly rehash, but just trying to make the communication explicit because I sometimes miss things: since it sounds like xd009642, tiberiusferreira, and myself would lean towards using Discord (now that I know about the size and variation of the Rust community currently using it, and the ease of switching between servers), I thought that using Discord would be the way to go.
If you do have a strong preference against Discord and for Zulip, we can make that change since no one else has communicated a strong stance. Are you expressing that, or alternatively is it the case that while you don't have a preference for Discord, you would be okay with using it with either our own dedicated server or a channel in the greater Rust server if the admins there give the okay for that?
Also, I agree that cross-pollinating with GameDev, especially regarding GPU compute wold be a positive! Once we have a decision made and chat medium set up, it would be great if someone (could be me) would make a post there inviting participation!
@deg4uss3r I was operating under the assumption that since we (as of yet) have not been assigned an official status as a language WG, starting our own discrete server would be the path forward (either on Zulip or Discord). If we were to be allowed to operate on either of those, then I'd fully support creating a channel or stream for this group under those umbrellas, depending on XR's response to your comment.
I think as an official applicant we could get a stream/channel (definitely on Zulip, maybe on Discord). A major concern of mine would be once we do became a WG and have a channel on places reintegrating the history of our server onto whatever platform we move official channels to. We will definitely want to have history at that point and rolling Discord server into a channel sounds complicated/very difficult with little permissions. Or worse we don't integrate because we have a large server and we continue seemingly unofficial or hard to find :(.
Another thing I think we should look into is a Code of Conduct for the group or at least once we restructure this GitHub group for the WG a mention and link to Rust's CoC and say all members/participants must abide by those rules to create a nice welcoming space for everyone.
I guess I should start a Dropbox paper or something with checkboxes for items edit:
todo list here: https://hackmd.io/@rs-ml-wg/r1Bq4XMPU
I was operating under the assumption that since we (as of yet) have not been assigned an official status as a language WG, starting our own discrete server would be the path forward (either on Zulip or Discord).
That is a good assumption, though we have created streams/channels for "non-official" groups before, I don't know if that's something we want to continue. I personally wouldn't have a problem with an unofficial wg-ml channel, but I'm not an admin on either platform, so you'd have to reach out to the admins for approval. It can't hurt to ask.
I don't want to bias the chat platform discussion, as it's more important that a group actively uses a chat platform rather than picking the ideal one. I will say one feature that Zulip has over Discord is email notifications, and reply by email, which are helpful for people who don't want to "hang out" on Zulip to be able to participate. Discord has features Zulip doesn't that might be more beneficial however.
@deg4uss3r Sounds great, looking forward to hearing what they say! Your concerns about rolling a full server into a single channel after a little make a lot of sense, and I agree that wouldn't be ideal.
I saw the to-do list, and all the items look good. I like the idea of linking to the Rust Code of Conduct and/or putting a slightly-modified-into-context Markdown version into the wg repository (when we have it). I put together a draft of it this morning that I can add whenever that I'd be happy to add when that repository gets created.
@LukeMathWalker I believe that @jbowles has been inactive on this platform for the past few weeks (I'm hoping everything's okay), so I think you're the only one who can add members/adjust permissions for the group at the moment.
@quietlychris I am not an owner. I'm still around and good :) ... crazy times though :). I've looked to add people on the to list by @deg4uss3r but I am unable.
As side note i have a few references to share:
- Data Analysis with Rust Notebooks : https://store.shahinrostami.com/product/data-analysis-with-rust-notebooks/
- https://github.com/guillaume-be/rust-bert
- https://github.com/guillaume-be/rust-tokenizers
@jbowles I think @LukeMathWalker invited is a members but not owners for the time being :)
Thanks for the links!
@jbowles I'm glad to hear it! I was just going off your commit history, and since you seemed to otherwise have a really consistent commit history, thought you might be otherwise engaged for the moment. I'm glad you're here, and happy that things are well, considering the state of things :) Thanks for those links as well! My background with NLP is pretty limited, so comparing those models to some of the design patterns for domains like image processing or NN is really interesting.
I hope no one minds, but I just added an initial commit of a wg repository containing a minimal README and a copy of the Rust Code of Conduct, which has been slightly adjusted per the rust-lang.org recommendations for external groups. Please take a look at both, and of course feel free to make any changes that you think might be necessary.
It currently has a list of group members in the README, but that can definitely be adjusted to its own document if that would be most helpful. I didn't add a "Vision" or "Purpose Statement" section yet since those hasn't been properly discussed.
feel free to make any changes that you think might be necessary.
Great job @quietlychris! Could you open the permission so members can push? I am not able to make any changes at the moment :D
@deg4uss3r Whoops, sorry about that! I just added everyone to a team that should have access--let me know if you have any issues!
Looks great! Have you had any word from the chat platforms to see if we might be able to get a stream/channel in one of them?