HTTP 1.1 Connectivity API - Source (aka Polling)

[jonaswre]

Discussed in https://github.com/eclipse-ditto/ditto/discussions/1541

^{Originally posted by jonaswre December 3, 2022} I have read the whole documentation twice.

Maybe I've missed something obvious, but in many scenarios its necessary to poll devices/gateways/services for data.

Is there any build in way to achieve this? I would prefer not have to run my own service that polls http endpoints and sends the result to a message broker.

Thanks alot.

[thjaeckle]

I think that is a good idea to use HTTP connection "sources" for configuring polling of HTTP APIs.

In order to do that, a "Consumer" for HTTP connection has to be created and started in HttpPushClientActor (currently, only a HttpPublisherActor is started there). How to create and start the "Consumer" role can be looked at from the other connection types. It e.g. involves creating a new class extending the abstract BaseConsumerActor class.

Regarding the scheduling you asked in #1541: you can use Akka's scheduler to schedule repeating tasks (like the polling). The polling interval should be configurable via the connection configuration (done in class HttpPushSpecificConfig).

The use of Akka's scheduler could look similar to (used in an Akka actor):

Object message = ....;
Cancellable cancellable = getContext().getSystem().scheduler()
   .scheduleAtFixedRate(initialDelayFromConfig, intervalFromConfig, getSelf(),
        message, context.dispatcher(), ActorRef.noSender())

This will send the message each intervalFromConfig to getSelf() (the own actor) - this message should trigger the polling of the HTTP endpoint.

[Abhijeetmishr]

@thjaeckle If I am not wrong this is related to message polling from queue service with the help of Akka Actor at periodic intervals or at certain intervals. I have prior experience working with Akka Actor with Aws sqs in scala. I will appreciate if there is more brief to this. Regards

[thjaeckle]

@thjaeckle If I am not wrong this is related to message polling from queue service with the help of Akka Actor at periodic intervals or at certain intervals.

@Abhijeetmishr not exactly .. it is about polling an HTTP endpoint at a configured location, HTTP verb and path .. I would suggest to create a new connection type "http-poll" for that which only supports "sources" - that requires quite a bit of understanding of the Ditto connectivity service.

If you are willing to try to work on that, I can write some stuff down how/where to start .. but this is a bigger journey :)

[Abhijeetmishr]

@thjaeckle Thank you for quick response will appreciate any kind of help. I would love to start and learn/explore on the go and I am totally available for any kind of dev related stuff either short or long term journey. Hoping forward to learn and collaborate with community. Regards

[Abhijeetmishr]

@thjaeckle Please provide any supporting doc so that I can start working on this asap.

[Abhijeetmishr]

@thjaeckle @jonaswre I have gone through this doc link looking for more resources to understand this topic and looking forward to explore ditto connectivity service. Pardon me if I am spamming I am just getting impatient.

Regards

[thjaeckle]

@Abhijeetmishr I am currently on vacation and cannot provide implementation hints. I suggest to try out and understand the current http connection type and have a look at the httppush code: https://github.com/eclipse-ditto/ditto/tree/master/connectivity%2Fservice%2Fsrc%2Fmain%2Fjava%2Forg%2Feclipse%2Fditto%2Fconnectivity%2Fservice%2Fmessaging%2Fhttppush

What this http push connection type is currently missing is an implementation for "sources". The to be added HTTP polling would be such a connection source.

[Abhijeetmishr]

@Abhijeetmishr I am currently on vacation and cannot provide implementation hints. I suggest to try out and understand the current http connection type and have a look at the httppush code: https://github.com/eclipse-ditto/ditto/tree/master/connectivity%2Fservice%2Fsrc%2Fmain%2Fjava%2Forg%2Feclipse%2Fditto%2Fconnectivity%2Fservice%2Fmessaging%2Fhttppush

What this http push connection type is currently missing is an implementation for "sources". The to be added HTTP polling would be such a connection source.

@thjaeckle Please let me know when you are available. So that we can start working on this asap if you have bandwidth. Regards

[thjaeckle]

@Abhijeetmishr you can start at any time - like I said I suggest to first grasp and understand the existing connections, especially the HTTP connection. Even after my vacation I won't lay out a complete plan how to implement what - if I would do that, I could also implement it by myself and be much faster.

[Abhijeetmishr]

@Abhijeetmishr you can start at any time - like I said I suggest to first grasp and understand the existing connections, especially the HTTP connection. Even after my vacation I won't lay out a complete plan how to implement what - if I would do that, I could also implement it by myself and be much faster.

@thjaeckle I am also not expecting complete layout there is no learning in that I have few questions if you can answer that, It helps in starting this.

What is the trigger point for http-push ? and how to understand the flow for http-push ? What is the use-case of http-push in our project and why we are planning to implement http-poll ?

Pardon me if I am asking anything silly. I will appreciate any reference/hint provided for above questions. I am beginner but I am ready to learn and do this at fast pace and I am absolutely positive about it. Regards

[thjaeckle]

What is the trigger point for http-push ?

Please read and understand the connections documentation: https://www.eclipse.org/ditto/basic-connections.html I cannot explain better than in the extensive documentation does.

and how to understand the flow for http-push ?

The same as for all connection types, however http-push currently only supports connection targets (also see documentation for difference between sources and targets).

What is the use-case of http-push in our project and why we are planning to implement http-poll ?

Yes, please explain what the use case in your project is and how http polling will help.

[Abhijeetmishr]

@thjaeckle I have gone through the documentation and explored about sources and target. But looking all at once seems overwhelming what are the baby steps I can take and then consolidate all the steps later on ? Regards

[thjaeckle]

To be honest, this is a really challenging issue to do. I currently do not have the capacity to guide you through this.

If you just seek for contributing, maybe a smaller issue would be better suited..

[Abhijeetmishr]

To be honest, this is a really challenging issue to do. I currently do not have the capacity to guide you through this.

If you just seek for contributing, maybe a smaller issue would be better suited..

okay no issue meanwhile I will keep on exploring and try connect dots. Thank you

[Abhijeetmishr]

@thjaeckle I don't understand which issue to pick next for contribution. Please help

[thjaeckle]

@Abhijeetmishr I currently do not see comparable low hanging fruit tasks as the 'ilike' was. All others are significantly more complex. Do you use Ditto and have suggestions what to improve? That is IMO always the best option to contribute. Improve something which you will benefit from by yourself.

[Abhijeetmishr]

@Abhijeetmishr I currently do not see comparable low hanging fruit tasks as the 'ilike' was. All others are significantly more complex. Do you use Ditto and have suggestions what to improve? That is IMO always the best option to contribute. Improve something which you will benefit from by yourself.

@thjaeckle Thank you for the guidance.

[thjaeckle]

Thinking about this issue I wondered how the polling would come to its thingIds to work on. So you would e.g. define:

• each 30 minutes, poll an HTTP endpoint via GET
  • but what if the HTTP endpoint already includes a "device id" or a "thing id"? where does that come from?

For other connection sources which are always reacting on an "outside" message, there are several ways to get a thingId:

• as part of protocol headers
• as part of e.g. protocol topic (e.g. for MQTT, Kafka)
• as part of the received payload

When a source now is triggered by a cron-job/polling, those information sources do not exist, so where do they come from? Ideas:

• one connection per thingId to poll for
  • does not scale very well - only usable for "small use cases"
• static configuration in the connection of a list of thing IDs
  • not very dynamic :/
• result of an internal search query, determining thingIds ... (e.g. searching for all devices of type "gateway" in Ditto, e.g. identified by an attribute)
  • could internally done via "streaming" the found thingIds in order to not need paging
• call of another HTTP API returning the list of ids first (this could also be Ditto's own search API endpoint maybe?)
  • but what about paging?

I think I tend into determining the list of thingIds to poll an HTTP API for via either static configuration of a list or by using an internal search. Do you have ideas on that @jonaswre ?

[thjaeckle]

This issue could be superseded by #1609