aws-greengrass-nucleus
aws-greengrass-nucleus copied to clipboard
aws-greengrass
com.aws.greengrass.mqttclient.spool: New persistent storage spooler(s) : sqlite spool / filespool
Feature Description Add new spoolers with persistent storage - either sqlite spool or a filesystem spool. These would be drop-in replacements for the InMemorySpool.
Use Case Where greengrass devices operate in conditions where they may be offline without network access for long periods of time (days, weeks) the InMemorySpool may not be sufficient for storing proxied IoT MQTT messages. Additionally, large MQTT messages will quickly fill up the default size of the InMemoryPool even when offline for short periods of time.
Proposed Solution
A basic filesystem spool would be the simplest to write, with each message stored as a file. But persisting many small messages could potentially fill up available inodes and have poor throughput performance.
An sqlite database spool would allow persistent storage to a single datastore.sqlite file, but with more CPU overhead.
The sqlite spool would be my preferred implementation.
Some limits should still be enforced on the persistent spoolers - potentially
- maxNumberOfMessages
- maxSizeOnDisk (harder to evalute for sqlite?)
Other Would this be something that the AWS team would work on internally / would you accept a PR if we implemented the feature?
- [x] :wave: I may be able to implement this feature request
Hi Kris, This feature is definitely on our roadmap. I don't have an ETA for you right now, but we're certainly aware of the case and we'll work to address it soon.
That's great :)
We'll be needing this for production in the next 6 months so we might start coding this and raise a PR, unless you know it'll be addressed before then.
@MikeDombo Sorry to chase this up, but is there a rough roadmap ETA? (and would you accept a PR for it?)
Yes we would consider a PR for this feature. I cannot share any ETA at this time, sorry.
We also need this feature. Would you now be able to share an ETA, by any chance? :)
@kris-sum Do you have any developments to share regarding this topic? This is a very useful feature i believe :) and the only workaround i found by far is using MQTT bridge (like in this training https://catalog.us-east-1.prod.workshops.aws/v2/workshops/5ecc2416-f956-4273-b729-d0d30556013f/en-US/chapter6-mqtt-broker).
As far as i see, Mosquitto broker has persistent option to write to the disk
https://pagefault.blog/2020/02/05/how-to-set-up-persistent-storage-for-mosquitto-mqtt-broker/#:~:text=The%20persistent%20storage%20in%20Mosquitto,true%20in%20the%20Mosquitto%20configuration.&text=If%20true%2C%20connection%2C%20subscription%20and,the%20location%20dictated%20by%20persistence_location.
But that looks way harder than native spooler configuration
@TamoshaytisV This is quite far down on my work backlog :( Hopefully someone else will be able to implement this!
Hi @MikeDombo - has there been any progress or updates on this at all?
Hi Tim, sorry but there are no updates here. We'll keep this issue updated for when work begins.
+1 to this being very useful. Since it's been a few months @kris-sum / @MikeDombo / anyone have an update here? I like the idea of a sqlite db with a max # of rows / size. Thank you :)
Does it worth mentioning this PR here? https://github.com/aws-greengrass/aws-greengrass-nucleus/pull/1277
However, i don't understand what is persistence spool plugin and where to find it to read about
+1 to this being very useful. Since it's been a few months @kris-sum / @MikeDombo / anyone have an update here? I like the idea of a sqlite db with a max # of rows / size. Thank you :)
Feature is still in our roadmap, no major updates or an ETA as of now, we'll keep this thread posted
Does it worth mentioning this PR here? #1277 However, i don't understand what is
persistence spool pluginand where to find it to read about
There's no documentation currently--when the feature is released, we'll update this thread with the new docs
+1 to this being very useful. Since it's been a few months @kris-sum / @MikeDombo / anyone have an update here? I like the idea of a sqlite db with a max # of rows / size. Thank you :)
Feature is still in our roadmap, no major updates or an ETA as of now, we'll keep this thread posted
Does it worth mentioning this PR here? #1277 However, i don't understand what is
persistence spool pluginand where to find it to read aboutThere's no documentation currently--when the feature is released, we'll update this thread with the new docs
Thanks for the update :)
Hey @jcosentino11! Any updates here? Sorry for the bother, but we're doing some development planning dependent on whether this would happen soon or not. Thank you so much!
We're hoping to get to this in the coming months, but cannot provide any ETA or guarantees at this time. This issue will be updated when there is more to share.
So happy to see that the spooler persistence feature is being merged into nucleus main! When is the next release?
So happy to see that the spooler persistence feature is being merged into nucleus main! When is the next release?
The change in nucleus allows discovery of the plugin---we're actively developing the plugin itself. I can't provide a specific release date but it's an active project, stay tuned :) Thanks for your patience!
Where can we keep tabs on the development of the plugin itself @jcosentino11 ? Thank you
There will be a repo once it's open source, but until then we'll keep this thread updated
Thanks!
On Mon, May 22, 2023 at 9:24 AM Joseph Cosentino @.***> wrote:
There will be a repo once it's open source, but until then we'll keep this thread updated
— Reply to this email directly, view it on GitHub https://github.com/aws-greengrass/aws-greengrass-nucleus/issues/825#issuecomment-1557532043, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABKCHIZDSLR2N4XQKP25P43XHOHLJANCNFSM4WOIHF3Q . You are receiving this because you commented.Message ID: @.***>
happy to announce the release of aws.greengrass.DiskSpooler https://docs.aws.amazon.com/greengrass/v2/developerguide/disk-spooler-component.html! (requires nucleus 2.11.0)