ModularSensors
ModularSensors copied to clipboard
EnviroDIY
Develop publisher to Hologram.io Data Router
Hologram.io and other IoT cellular SIM providers (such as Soracom.io) have flexible data routers designed to directly receive data from the IoT field device and forward it to one or more endpoint destinations on the cloud or an application server (such as MonitorMyWatershed.org)
A ModularSensors "Publisher" to Hologram's Data Router would give a lot of flexibility and provide enterprise-class uptime for receiving data. This could represent one small step toward over-the-air (OTA) device management.
Here’s documentation on how to do it for Hologram.io: https://hologram.io/docs/guide/cloud/data-router/
Here’s how to do it for soracom.io:
- Funnel https://developers.soracom.io/en/docs/funnel/
- Beam https://developers.soracom.io/en/docs/beam/
see description below of Funnel vs. Beam.
Compared to Beam
While Beam and Funnel both provide similar functionality in receiving data from your device and sending it to a specified endpoint, Beam is more general purpose and allows you to mix and match various incoming and outgoing protocols, as well as specify endpoints other than cloud services such as your application server. While Beam provides flexibility with extended protocol and endpoint support, it also requires additional configuration, even when using it with cloud services. On the other hand, Funnel provides a greatly simplified configuration for applications that leverage cloud services. Funnel takes care of most of the forwarding protocol configuration required for certain cloud services, allowing you to simply specify only the forwarding destination and credentials in order to start sending data.
cc: @everhamme; @gcutrell; @SRGDamia1
From @everhamme:
BTW.. Soracom BEAM can provide a nice http/https bridge to strengthen security.. Soracom has a closed VPN connection to every carrier they support.. so data could flow via HTTP through this tunnel and then Soracom BEAM can support a cross over to HTTPS to prevent keys from being exchanged over the public internet… see below..
Offloading Encryption
When a Soracom Air SIM device connects to a cellular network, data transmission is secured through SIM authentication and closed carrier infrastructure, and passed to the Soracom platform over dedicated direct lines. Since communication between a device and Soracom is secured, you can utilize lightweight protocols such as HTTP or MQTT to reduce cellular data usage and power consumption, and then perform encryption in the cloud for protecting data between Soracom and your network environment.
Per this thread Dan already set up a publisher to Holograms cloud socket api: https://www.envirodiy.org/topic/ubee-r410-setup-issues/
That’s a long thread to follow! Can someone writeup an easier to follow use case?
From: Sara Damiano [mailto:[email protected]] Sent: Wednesday, January 29, 2020 5:35 PM To: EnviroDIY/ModularSensors [email protected] Cc: Ed Verhamme [email protected]; Mention [email protected] Subject: Re: [EnviroDIY/ModularSensors] Develop publisher to Hologram.io Data Router (#305)
Per this thread Dan already set up a publisher to Holograms cloud socket api: https://www.envirodiy.org/topic/ubee-r410-setup-issues/
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/EnviroDIY/ModularSensors/issues/305?email_source=notifications&email_token=AC3NWGWTAX3KJZ6AF3XNGBLRAIACHA5CNFSM4KNMNYG2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEKJACEI#issuecomment-579993873, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AC3NWGRHWBWEETZ4HBY4YTDRAIACHANCNFSM4KNMNYGQ.
I'm sorry, no, that thread wasn't about a particular use case, just someone that mentioned doing something similar. There's nothing else related in the thread; you don't need to read it.
I can't find his fork of the library, so he must not have shared it here.
Reading up a little, both services are just data forwarders - so you send data to them with a presumably smaller metadata load and/or lower security level than your desired service requires and then you use their web tools to fill in the gaps.
So for data to MonitorMW the path might be something like.. send data to beam/data-router as "raw" tcp information in JSON with minimal key headers - could be a single number indicating the order or a variable in the array or a "compressed" version of the UUID. Within the Soracom or Hologram interfaces, you set up a translator that decompresses the UUID's and adds HTTP headers. Then MonitorMW receives a complete http request and appends the data appropriately. The advantage would be use of less cellular data on the logger side because of reduced http overhead and compressed uuid's. Of the two services, the hologram tools seem to be much more amenable to this approach - Soracom's main advantage is security, not compactness.
I doubt it would take more than a few hours to get data going from ModularSensors to hologram. It's a very light lift.
I don't know how many more hours it would take to set up the forwarding accurately on the hologram website. For multiple loggers I think you'd very quickly end up with a nightmare of look-ups translating between logger x - variable y and site a - uuid b without any tools on the hologram end to manage it. After all, you'd basically need the fully fleshed out equipment extension of ODM2. Given that immediate headache, I'd guess it would be easier to add an mqtt broker and compressed-hash-translator into MonitorMW than to write a tool that could access the hologram API and apply all the look-ups you need. And without said tool - you're regressing back to managing data in individual excel sheets that tell you what's what. So.. @aufdenkampe do you have a different use case?
Use case is providing cloud agnostic equipment so data routing can happen at sim level as opposed to device level. Independence of hardware and cloud software is the goal to ensure future operation of sensing network as cloud platforms come go (but sim provider stays)
Ed Verhamme Project Engineer 501 Avis Drive, Ann Arbor, MI 48108 Office: 734-332-1200tel:734-332-1200 Direct/Cell: 734-681-0577tel:734-681-0577 Email: [email protected]mailto:[email protected] http://twitter.com/eddiegreatlakes LimnoTech www.limno.comhttp://www.limno.com/ Water Environment | Scientists Engineers
-------- Original message -------- From: Sara Damiano [email protected] Date: 1/29/20 6:44 PM (GMT-05:00) To: EnviroDIY/ModularSensors [email protected] Cc: Ed Verhamme [email protected], Mention [email protected] Subject: Re: [EnviroDIY/ModularSensors] Develop publisher to Hologram.io Data Router (#305)
I doubt it would take more than a few hours to get data going from ModularSensors to hologram. It's a very light lift.
I don't know how many more hours it would take to set up the forwarding accurately on the hologram website. For multiple loggers I think you'd very quickly end up with a nightmare of look-ups translating between logger x - variable y and site a - uuid b without any tools on the hologram end to manage it. After all, you'd basically need the fully fleshed out equipment extension of ODM2. Given that immediate headache, I'd guess it would be easier to add an mqtt broker and compressed-hash-translator into MonitorMW than to write a tool that could access the hologram API and apply all the look-ups you need. And without said tool - you're regressing back to managing data in individual excel sheets that tell you what's what. So.. @aufdenkampehttps://github.com/aufdenkampe do you have a different use case?
You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/EnviroDIY/ModularSensors/issues/305?email_source=notifications&email_token=AC3NWGUKV6U4IAYLXD26DTDRAIIEHA5CNFSM4KNMNYG2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEKJFRPI#issuecomment-580016317, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AC3NWGUQVBJGVI43MO7UOILRAIIEHANCNFSM4KNMNYGQ.
Well, as I said, I think hologram would be pretty easy and Soracom not especially difficult.
If you're talking about the longevity of components, though, I would say the "device" running ModularSensors is probably the most short lived of all - For any of these Arduino loggers I would expect to have to replace the data logger sooner than the cloud software. Obviously I've invested a tonne of myself into this library and think it's fairly decent, but if someone was going to ask me if they should use it for a remote sensor that would take a great deal of time or effort to access for reprogramming or service I might recommend a different platform.
Thanks Sara. We're approaching 100 plus devices deployed between cellular and lorawan and its great to have modular sensors and other options available.
Ed Verhamme Project Engineer 501 Avis Drive, Ann Arbor, MI 48108 Office: 734-332-1200tel:734-332-1200 Direct/Cell: 734-681-0577tel:734-681-0577 Email: [email protected]mailto:[email protected] http://twitter.com/eddiegreatlakes LimnoTech www.limno.comhttp://www.limno.com/ Water Environment | Scientists Engineers
-------- Original message -------- From: Sara Damiano [email protected] Date: 1/29/20 7:54 PM (GMT-05:00) To: EnviroDIY/ModularSensors [email protected] Cc: Ed Verhamme [email protected], Mention [email protected] Subject: Re: [EnviroDIY/ModularSensors] Develop publisher to Hologram.io Data Router (#305)
Well, as I said, I think hologram would be pretty easy and Soracom not especially difficult.
If you're talking about the longevity of components, though, I would say the "device" running ModularSensors is probably the most short lived of all - For any of these Arduino loggers I would expect to have to replace the data logger sooner than the cloud software. Obviously I've invested a tonne of myself into this library and think it's fairly decent, but if someone was going to ask me if they should use it for a remote sensor that would take a great deal of time or effort to access for reprogramming or service I might recommend a different platform.
You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/EnviroDIY/ModularSensors/issues/305?email_source=notifications&email_token=AC3NWGQA3O3PCMTSXNSN6KLRAIQLTA5CNFSM4KNMNYG2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEKJJ5GQ#issuecomment-580034202, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AC3NWGSDJ256CSDQPPSZORTRAIQLTANCNFSM4KNMNYGQ.
It's been a while since I have checked in on the EnviroDIY world and just noticed this issue. I'm the Dan that got Hologram working. If there is still interest I will share the repo. It's not too pretty but it works.
Hey Dan thanks for the work and sharing your PR -https://github.com/EnviroDIY/ModularSensors/pull/323 and files https://github.com/EnviroDIY/ModularSensors/pull/323/files#diff-a20444ce1820c603ab9865c56e2e8340
I'm just wondering if you have an example of what you put in Hologram for the forwarding from hologram to ?( MMW).
I've got a reliable data delivery, https://github.com/EnviroDIY/ModularSensors/issues/194 working, and just wondering if you've got a trace for how acknowledgements (successful POSTs) work through hologram. That is do you have a feel, or trace, for what constitutes a successful delivery of a message to the end datastore.
Looking at the API https://www.hologram.io/references/embedded-apis it says "On success, the server will return the text [0,0] and then close the connection. On error, the server will return a different message, or it may close the connection with no response."
Form the Mayfly the no response timeout of 10secs, is of course is encoded as HologramPublisher.cpp#254 responseCode = 504;
@dnowacki-usgs, thanks for sharing your work back to us via PR #323!
I've been long interested in trying this, but hadn't found the time to dig in. I'll be very interested in looking over your PR!
I agree with @neilh10 that it would be great to have some info or even screen-shots of how you configured the forwarding on Hologram. Where & how do you forward your data?
I'm pretty sure on Hologram the APIs are automatically enabled (could be wrong on that). I then use the JSON API via a custom Python script run via cron to retrieve the data from Hologram and save it locally on a server as netCDF. Here's an example.
As for checking to see if the connection is successful, I don't do any. Every once in a while I lose a reading when the modem doesn't connect. Most of the time we are using sensors that are logging internally in the field anyway (and expose data redundantly via SDI-12, which is how I'm reading data from the instruments onto the MayFly) so it's not a big deal if we have a transmit failure. But for those instruments that don't log internally, ensuring connection success would be a huge improvement.
Hey thanks. Sounds pretty simple. Interesting that hologram is saving it so it can be polled for. The new reliable delivery functionality that I have saves readings if there isn't a completion recorded, and then pushes them later.
Looks like your software polls for all readings, so I wonder if you see that hologram keeps all readings, or is it only the last reading posted from the mayfly.?
Hologram retains all of the readings, or seems to. The script reads the latest date contained in the netCDF and only requests data newer than that in the API call.