CodeIgniter4
CodeIgniter4 copied to clipboard
codeigniter4
WIP Queue class prototype (fix PR #347 GPG-sign)
Description
#117
I am really sorry for having kept you so long. 2 years. It's too long. I apologize sincerely.
This is fix #347 GPG-sign and some test codes. I don't fix main codes, so perhaps it is low quality as beta-1.
Checklist:
- [ ] Securely signed commits
- [ ] Component(s) with PHPdocs
- [ ] Unit testing, with >80% coverage
- [ ] User guide updated
- [ ] Conforms to style guide
Does this replace the earlier PR? This should be for the feature/queue branch - fixed. Some of the class positioning is out-of-sync with our current adapter patterns (see https://forum.codeigniter.com/thread-72997.html). Your config data & placement feel awkward.
This won't be in 4.0.0, but an upcoming release instead. We are not likely to invest too much time at the moment, as we concentrate on 4.0.0.
You are gettig all of the cruft commits because you didn't synch your develop branch with the main repo, i think. I have synched the feature/queue branch on the ci4 repo, so that you can hopefully make a trivial change in your work and push that to your feature branch, and have this turn out "clean".
There's no urgency on this one. It won't be included in 4.0, but in 4.1.
sorry. it took a little time to read other feature handlers. i rewrote the code.
What's the plan with this? I'm pretty interested in queues and this looks like quality code. Does "Blocked" mean intentionally don't want this for RC1? If so is that a case for starting a 4.1 or 4.0-RC2 branch?
The PR is in the "blocked" column because nothing is happening with it. It isn't being actively worked on, but we don't want to forget about it. It can always be worked on & not merged. We haven't talked yet about longer lived branches beyond master & develop. This is a new feature, so shouldn't have any breaking changes, so it is possible that it might be a candidate for 4.0.0 or 4.0.x
Thanks for the update. @noldor are you still around? I see no user activity since February.
I was looking at making a Task Scheduler in a future project of mine and I stumbled onto this. I synced the branch to develop here ( I'm not sure if creating a new branch and merging 'develop' with it was the proper way to do that, please let me know if I should have done it a different way).
I'm reading through noldor's code right now, do you have any comments, or ideas on how to move forward with this?
Some ideas I had so far after also reading the previous PR thread, please let me know what you think:
-
We restrict Queue to just that, a simple service that interfaces with a number of messaging services/handlers (Redis, RabbitMQ, DB etc) and allows you to add data in a queue to be retrieved in a specified order (maybe also allowing for batch insert/retrieval). Pretty Much what noldor is doing
-
We create a second Job Runner/Task Scheduler service which uses the Queue service above to process the data in the queue. This will probably use some form of external scheduling, like cron running a spark command every minute to trigger it. Some internal Rate Limiting might be usefull
I think separating the two services allows for greater flexibility.
Example uses:
-
Mailing/Newsletter service. Enter a list of client data into a queue called "newsletter". A create a Job that goes through that queue, takes each client's data and sends a personalised newsletter using their details to their email
-
Product price check. Say there's a website were multiple merchants can add their products to sell. That website provides an API that allows you to check the lowest price for a specific product SKU. You want to check daily whether you have the lowest price there. You add a list of products you want to check to a Queue and then setup a job that goes through the product list, fetches the current price and re-adds the product to the queue updating 'exec_after' to now()+1d
-
General Maintenance: Tthere's a list of things you want to be doing on a specific interval X. EG: Empty some cache folder, set members as inactive if they haven't logged in for some time, check if any composer packages need updates. Add those tasks in the queue having their "data" be some sort of ['route/to/controller'=>['data1','data2']] notation. The scheduler goes through the tasks that haven't been run in the past X time, passes each task's data to the respective controller and if successful re-adds it to the queue updating its 'exec_after' value
I have not had a chance to revisit this in quite a while. And a Task Scheduler should definitely be separate from the Queue. I've gotten the barest of starts on one and if you wanted to jump on development of that, I would gladly accept it. My free-time is a bit scarce currently, and most of that seems to be reviewing commits.
I added my list of items I think it should eventually do on the Readme.
Once finished, this will be an official CI4 project, that will live on its own but also eventually get merged into CI4 core.
And, yes, I think the Task Scheduler needs to happen before the Queue gets revisited.
Feel free to jump in over there and ask questions, or start a thread on the forums (though that takes me longer to see...)
I needed a basic database queue interface so I started work on this: https://github.com/colethorsen/codeigniter4-queue which took the basic structure that was here and turns it into something more usable at least for the local database queue.
I've also been working with the base of tasks setup, there are definitely some ways that they would work together.
We're still keen to offer this in the core. If you get to the point where you want to build on this for the framework itself please share your work!
@MGatner if you look at the code it's structured specifically so that it could easily be ported straight into the core, it's in its own repo purely so it can be easily included in projects without being in the core for the moment. It's basically structured the same as the Tasks repo.
@colethorsen Could you please add a license to your repo? It looks like a lot of the content is straight from Noldor's PR but if there is any original work it will be unusable since your repo has no license.
EDIT: Actually, I'd like to chat some more about the changes you've made. Are you on the CodeIgniter Slack? Or some other place we could discuss? PM on the CodeIgniter forums (if you're not on the Slack) and we can set something up.
Hey @colethorsen I saw your other PR... if you're around could you check out my messages above please? Keen on getting Queue rolling.
@MGatner Happy to talk about it, I'm not on the slack (can't sign up as the link is broken) https://codeigniter.com/community and the forums also seem to be down, as the sessions table has crashed.
All. This is great. I had not checked the PRs before and @lonnieezell tuned me into this.
I had independently cobbled rabbitMQ controllers together to fulfill this same underlying purpose using lavary/crunz as a scheduler. I felt I was on the bleeding edge (or down at least a rabbit hole) to some degree and got in touch with Lonnie when I reread the CI forum and was following links through to his scheduler in development.
Philosophically from a CI design perspective I like the idea of "one single unifying Model to handle all types of persistence" (Lonnie's accurate paraphrasing of my position). I hope he does not mind me sharing that his experience is "that it doesn't work as well in practice as it sounds like it should" which I can well appreciate. Unpicking something that is already working well is complex.
I believe that if I am reading the code of this PR correctly that it is using queues for management of the db commits and results (similar to what I was able to achieve with a controller). Effectively an interface that handles the db work.
In my minds eye, embracing queues in CI would go deeper than that. To reduce the complexity of Models, queues would be integrated into its core. There would be consistently named and purposed queues for inputs and outputs that the Model interacts with. Some default queues and standards would be put in place (though fully configurable) with a first objective to satisfy all current db model functionality. The complexity of the db interface would be moved out of the model to reside at the end of these defined queues themselves (eg. the db wrapper class interacts with queues not the model itself).
So to simplify Model functionality and extradite embedded db functionality therein, the Model might interact through queues relating to commonly used functionality - authentication, headers, footers, query_builder (instruction validator), interface builder (forge), output(read, write, delete requests - able to deliver to separate queues), results.
If I wanted to build another interface (email, sms, RESTful API, etc) it would sit on the opposite end of corresponding Model queues. eg num_rows would provide queue depth, result_array, or result_object would drain the result queue and present accordingly. The model would have consistent core functionality regardless of the interface. Core model functionality would be queue interaction based (not db). Very interface specific extra stuff not otherwise common could still be achieved by model to a purposed queue for that purpose.
The engine executing the (eg db) wrapper at the opposite side of Model queues might be the same (originating) instance of CI, a standalone instance of CI dedicated to interface(s) interaction ie. configured and permissioned, or some standalone execution engine or api entirely. As long as it responds to the originating Model queues in a known way it is good. Combined with a scheduler this allows modular builds, performance and scaleability.
Use case: I currently have three CI instances that on separate schedules load queues. A completely separate CI instance drains these queues and interacts with corresponding dbs and posts responses to queues as required. Only one of my CI instances needs to interact with db's and I can resource it accordingly.
Theoretically I might wish to have a single user authentication system that interacts with all my CI instances. I would hope to use an interface agnostics CI Model concept to accomplish this in the future too!
IMHO, The Model needs to be extracted from its db origins and I think queues/message bus interaction is a strategic way to do it. It does add an extra layer, but I think RabbitMQ is pretty solid as a default queue manager. Strengthening core CI MVC functionality, essentially via "environment" configuration that define queues (interfaces) would be powerful.
I know this is beyond the scope of the current PR but what are thoughts from people that know better than I ?
@simmonspaul Thanks for your detailed description and proposition! Just to be super clear I understood you, you are suggesting an implementation of queues that uses CodeIgniter\Model (or CodeIgniter\BaseModel) to give a common set of familiar methods? If that's the case I would be opposed, not from an idealogical stance but a logistical one. Our Model classes are a bit of a mess right now, and I would rather see new functionality started separate than piling it on.
I will also add that our tea is pretty strapped, and we are looking at (in this PR and Cole's subsequent work) an almost-ready implementation of Queue. I'm not opposed to an entirely new approach but it would need a champion willing to carry it through in a short amount of time. If that champion is you I would be happy to discuss more or review a PR! Either way I'm grateful for your presence here - I have taken on getting Queue into core but I have zero experience with raw queues or Rabbit so someone with some on-the-ground knowledge would be helpful.
@simmonspaul @MGatner this stuff is heavily based on other queue integrations such as Laravel, which I think do a pretty good job of the implementation. So in 1 way you'd be making it similar to how the CodeIgniter model works for consistency, and in the current way it follows the design patterns of other queue/job systems for a different consistency.
Given that queues work fairly differently from database tables, I'm not sure I see making them into existing models as all that beneficial, it would either require a lot of additional functionality in the core model, or a significant extension, and I'm not sure where the benefit of them being the same codebase is. If you wanted to update naming conventions so the queues and models were closer, but still separate that'd probably be good, off the top of my head changing send to insert/save/update and fetch to find, perhaps a methodology for querying the find similar to how you query a database?
On a further tangent, the way this system lets jobs dispatch themselves is kind of handy, and may be something worth investigating for entities/models, allowing an entity to persist itself.
@MGatner
Our
Modelclasses are a bit of a mess right now, and I would rather see new functionality started separate than piling it on.
I'd be interested in hearing more about this, I've always liked the limited scope of the CodeIgniter Model (going way back in CI) which allows for it to be easily extended with additional functionality as necessary. Having said that I have a bunch of code in an extended model for managing joins/relationships between data, and making it faster to pull related data if thats where you're looking to go.
Just to be super clear I understood you, you are suggesting an implementation of queues that uses CodeIgniter\Model (or CodeIgniter\BaseModel) to give a common set of familiar methods?
@MGatner Yes, ultimately that would be the objective
we are looking at (in this PR and Cole's subsequent work) an almost-ready implementation of Queue. Given that queues work fairly differently from database tables, I'm not sure I see making them into existing models as all that beneficial, it would either require a lot of additional functionality in the core model, or a significant extension, and I'm not sure where the benefit of them being the same codebase is. Our Model classes are a bit of a mess right now, and I would rather see new functionality started separate than piling it on.
@MGatner, @colethorsen I've been using CI for sometime and learning as I go. The power and simplicity of using CI's MVC framework is what appealed to me and has drawn me in. However I realised that I often had requirements that needed to be satisfied outside "MVC" through extensions because model was not agnostic but db centric. I believe that philosophically this is an unnatural constraint. In the long future new users should be able to use core MVC (indeed Entities) to interact with all sorts of endpoints in a consistent way.
I hear loud and clear that it is:
- too complicated to rework CI\Model, its a happy mess that works great and for what it was designed, those that know it understand and rely on how it works
- Testing the idea of embracing queues more deeply shouldn't stand in the way or leapfrog on the back of PRs already in flight
Practically I think this would be a fully parallel extension eg. CI\Mold which would deliver CI\Model functions relating to $table and $builder ("Working with Data" functionality) to queues. (We could also create command pseudonyms as @colethorsen mentioned.) * The queue handlers in this PR could be leveraged for queue interactions. Success would be:
- populating a set of queues in response to "model" commands and listening to corresponding queues for a response
- CI\Database directories not being referred to
- timeouts occurring on the listeners! (as a responder has not yet been created)
The next part would be a Mold Database Responder which would listen to $table and $builder queues that had been populated and to use the CI\Database code to return a result to corresponding output queues. ie The corresponding mirror side of CI\Model functions responses.
The two parts working together should produce the same result as using CI/Model. This proof of concept is what I refer to as the grunt work.
The real brain power and thinking that would need to occur over time is in the science and reengineering of Model.
- defining purposeful common queue definitions for use regardless of interface/end point
- keep configuration easy
- establishing standard message formats that achieve Model responses and beyond
- establishing default permissioning/prioritisation/timing of messages and interactions
- Making it easy to add different types of responders in the future
As a legacy, this solution would remain backward compatible (forever). Perhaps db connections would be environmentally configurable to connect directly without MQ as today (would pass straight through to existing CI\Database calls).
On that basis I would hope that this functionality would prove its worth to eventually replace Model.
Please more thoughts...
(I can eventually get around to grunt work if this is potentially in the communities interest. I am more hacker than programmer, so I would need some guidance to herd my own cattle.)
Thanks, that explanation helps a lot. Let me say right off: CodeIgniter\Model is a database class. MVC has shifted through the years as frameworks have adjusted to modern web app needs. If you want to look at CodeIgniter through an MVC lens's I think it would be helpful to rename (in your head) our Model classes to "Database Entity Factory". The concept of "MVC model" is a larger category of services, libraries, and data representation than what we have explicitly named.
That said, I'm not sure if there is any merit to creating a universal "MVC model" class. What a Controller does with a Queue response is going to be very different from what it does with a database row. To be sure some classes (like Entity) will have multipurpose value as data representation, but that's different than trying to streamline your factories.
As for class methods, I follow the logic of naming conventions (e.g. Model "finds" so should Queue "find" as well?) which is fine as long as it isn't forcing illogical concepts. Pushing and popping from a Queue makes much more sense to me, with a "queue" being a logical "line of objects".
@simmonspaul @MGatner I agree, there is very little commonality between what a Queue does and what a Model does, and shoehorning queue functionality into a model to attempt to "simplify" the design pattern doesn't seem logical, this seems like it would muddy the waters of how queues generally work (not to say the current way is the definitive right way) but with little added benefit and a whole lot of added complexity I don't really see why this would happen.
Even the way most queue systems are designed and named within their APIs doesn't match up with a database, so renaming (send, fetch, receive etc) to find, insert, update takes the queuing system away from the way that people understand that queues work. My preference would be to treat queues like queues (in the way they are designed, and the way they are named) as opposed to creating a whole system with a different naming convention that would cause added complexity.
@colethorsen @MGatner Thanks. Your thoughts, ideas, and consensus was exactly what I was looking for.
The concept of "MVC model" is a larger category of services, libraries, and data representation than what we have explicitly named. little added benefit and a whole lot of added complexity I don't really see why this would happen. I can readily accept both of these arguments. Model is a powerful db class. I'm far enough up the curve that it doesn't really much matter to me that it does not handle other types of data. I guess as my use cases and design have become more modular, I felt detached often having no 'model' at all after having relied on its usefulness for so long...
Both the Queue and the Task scheduler are installable via composer, and usable. The queue relies on the task scheduler, so it have to remain a separate install until the queue functionality is directly added. For the time being I've been using both as separately installed packages.
Alternatives:
- michalsn/codeigniter-queue
- colethorsen/codeigniter4-queue
- arashsaffari/queue
- masrodjie/codeigniter4-queue
For those looking for a solution to this, @michalsn has created a good implementation you can use right now.