StephenOTT
How to contribute
I'm really interested in your project.
I've been working on something similar based on the zeebe simple-tasklist, but your design is much better.
How far along is this project? Is it a serious project, or were you just prototyping?
I'd love to contribute but I'm not where to begin - I attempted to do mvn test but it just seems to hang waiting for some external connections it appears.
The first phase was to scope the components out and get understanding of the different unknowns. That was done.
Current code works as a long polling zeebe client that can collect user task jobs and create the entities in the DB and then have a user complete the task using a formio form and it will validate the submission, followed by completion of zeebe job.
First phase was built in vertx. But with updates to micronaut and their updates for reflection free aspect annotations(Such as their bean validator), I will be migrating the code to micronaut and kotlin. Want to reduce a bunch of boilerplate code that micronaut and kotlin will take care of.
The migration is just a code migration, it's all the same logic that's already in code. Hoping to get back to it in the next few weeks.
The test is just a placeholder to generate the Mongo db and zeebe engines.
What sort of use case do you have for the user tasks?
Great work Stephen. I'm building a stack for a client using a Gentics Mesh CMS, a custom notification/chat service built on Kafka, and BPM. The ideal solution would allow the business users to modify the workflows to generally change how the application functions. Registering workers for task types in Zeebe seems like a good approach for this - it just depends on how granular we want to get with functionality, or how generalized we can make the workers. The more generalized I imagine the approach, the more complex the task configuration gets in the workflow, defeating the goal of simplification for the business users.
This is especially the case in user tasks - for example there is "content publishing", in which an event will start a workflow that has a user task, the task should reference the content item being reviewed, then the user can complete the task by submitting a new status and/or comments. With the current approach, the workflow developer needs to embed the form definition as a json payload in the task headers. I like the idea of the form id referenced in the task - I think that's more reasonable for the business folks.
Another use case is more generalized - they want to be able to assign a task to a user with a deadline. The user would complete it by adding notes and/or a status.
Although not requiring BPM, another use case influences the technology decisions - they want to be able to design forms, make them available to the users, collect the data, and report on it. This is why I was looking at FormIO.
I know Camunda is great for the user-task stuff. But if I tell the team we're going to use 2 BPM engines, or that we're switching engines, they're going to lynch me =\
I've got lots of experience in Java/Spring, but only recently started learning about vertx [because the CMS also uses this], and have no experience with Micronaut or Kotlin. That being said, I'm a pretty quick study - and would love to contribute in some way.
Out project depends on a robust user task solution, so I'm highly motivated!
Thanks for the details. That helps a lot to understand.
So from your description I am seeing many cases of long running processes (processes that could be alive for hours, days, weeks, months etc).
Take note of this limitation in zeebe at the moment https://github.com/zeebe-io/zeebe/issues/3254. It prevents the ability to migrate a active process instance to a new version.
For your use case the above i think will end up being a problem because users will only be able to modify workflows for the next created instance. But won't be able to modify active instances. Example: you generate a task to review and approve a doc/node/page by the end of the month. If you decide 1 week into the month that you need to update the workflow, you can't effect the already active instances.
Another consideration for you guys is the level of users you are expecting to model the bpmns and manage the various configurations: it generally takes a skilled user to do this. A pattern I have used with clients is to create process definition templates with expressions: where you create common process patterns that use expressions ${...} throughout the model to essentially make it configurable at runtime. A example model could be "start form + approval step + approved or denied step".
This way users can configure a pattern rather than inventing the pattern from scratch. You still have the advanced ability to build from scratch, but the patterns help the typical content writers in a CMS type env compared to devs building a orchestration.
Let me see what I can move around to make some time this week to get the micronaught stuff started to give you something to start building from.
Are you building your UI in gentics?
Take a look at the the models here https://github.com/StephenOTT/OpenC2-Orchestrator/blob/master/README.md#openc2-orchestrator-with-worker-using-a-proxy-network-to-limit-internal-network-exposure. Should give you some ideas on possible ways to structure your workers, especially if you end up building the workers in vertx
Thank for the insight! In-flight updates will be an issue, I'll need to manage that expectation with the client.
Good advice about the workflow templates. They're planning on hiring developers for the handover, and have asked me what skillsets they should look for - so maybe we can get ahead of this one. Still - templates are good for consistency and efficiency, even for the seasoned folks.
A web-based BPM editor is out of scope for the current phase, so we'll be using the Zeebe modeler desktop app, and have an interface to download/upload the workflows. I was thinking we could store the BPMN in gentics, and have an eventbus listener deploy the workflow. The rest of the UI is custom React - the gentics UI will probably be hidden from everyone but admin.
I'll need to digest the OpenC2 references - this is new to me. We've also implemented the Kafka-connect sink/source-worker for Zeebe - https://github.com/zeebe-io/kafka-connect-zeebe Trying to decouple the components more. For example, we're feeding the gentics eventbus messages into Kafka so another service can start a workflow based on an event-to-workflow mapping.
I'll get familiar with Micronaut now - it looks interesting.
A plugin to the Zeebe modeler to upload the BPM, using the zeebe-node client, would be easy to write. The main blocker for me just doing it is knowing how to add a plugin to the modeler.
Update: let's see if I can get somewhere with this https://github.com/zeebe-io/zeebe-modeler/issues/137
@jwulf what would the plugin solve for this context? Good chance that you would be uploading through a proxy system to manage login, auth and authz rather than directly to the cluster. (This is the same reason for the management server deploy worker capability https://github.com/StephenOTT/Quintessential-Tasklist-Zeebe/blob/master/README.md#deploy-workflow
@jdbranham update for you, I should have initial release with basics in place tomorrow
@jwulf what would the plugin solve for this context? Good chance that you would be uploading through a proxy system to manage login, auth and authz rather than directly to the cluster. (This is the same reason for the management server deploy worker capability https://github.com/StephenOTT/Quintessential-Tasklist-Zeebe/blob/master/README.md#deploy-workflow
In dev you can ignore that, which is sufficient for an MVP. On Camunda Cloud you use M2M OAuth, so all you need are client creds and it just works.
little behind but conversion has started here: https://github.com/StephenOTT/Quintessential-Tasklist-Zeebe/tree/micronaut
Update for you on migration:
- Form server and validation logic has been converted and tested
- User task controllers, storage and logic has been 90+% migrated. Few smaller updates around the specific logic of actions like claim/unclaim/assign. But base is there is examples of the pattern. I also added form versioning so you can work with multiple schemas against a single form key so you can version your implementations.
- Zeebe manager server to deploy a workflow has been migrated
- Zeebe start workflow has been migrate but needs more testing.
- Zeebe testing has been moved to TestContainers so make sure you have local docker to run your tests against the Zeebe clients.
Just mainly cleanup and adding in the QoL adjustments.
Update. All the parts have been added for recreation of end to end. You can deploy workflows, start workflow instances, the worker will generate user tasks in the db, add forms and form schemas, submit for task completion using form submission data, etc
Everything is rewritten in rxjava. And the worker is a custom worker wrapper using the zeebe java client. The worker is rxjava and will do batch + repoll as the pool in the batch is available. So example: say you have a python worker that executes scripts that take 1min to complete and you poll and have the batch max jobs set to 10: on first poll you find 1 job, and start execution, in parallel the worker will repoll for 9 jobs. There is the user task worker, and an example python executor worker that generate from the same GenericWorker
User task form submissions that pass form validation will have the job completed in zeebe and the user task in the db updated to a completed state.
Round trip submission to complete a user task in dev modes of full stack on ~low-powered laptop is sub 600ms