ModularSensors
ModularSensors copied to clipboard
EnviroDIY
HAL Discussion
This is a discussion on HAL and I share it in the hope it remains open for identifying what works to abstract and is easy. (2019Aug11 updated to reflect code changes)
For software targeted at multiple hardware platforms, its typical best management to identify a "Hardware Abstraction Layer" - or a HAL - https://en.wikipedia.org/wiki/Hardware_abstraction. Arduino's success to some degree is providing basic level HALs that really simplify an embedded processor - https://playground.arduino.cc/Code/HardwareAbstraction - captured in simple APIs. The HAL discussion is always an evolution of extensibility and different tradeoffs.
Modular Sensors (MS) while implemented on an initial family of AVR processors and the Mayfly , is providing capability for expansion with other platforms (mostly ARM SAMD21 & SAMD51 and derivatives but could be others with low cost cellular wireless ). The core of MS is abstracting different equipment interfaces as extensible classes. Hurrah!!! The ProcessorStats.cpp is one example of a simple HAL.
For this discussion the HAL is different than a platforms mapping to hardware as done with the arduino framework variant.cpp.h
The following are areas that I believe could be considered for abstraction for future readability:
The "wall time clock" is platform and processor dependent - for the Mayfly its encapsulated in a battery backed up reliable RTC DS3231 (https://github.com/EnviroDIY/ModularSensors/issues/236) - for other processors the functionality is distributed. It would be valuable to have it extensible.
The board LED(s) could represent a user view of the state of the system. A USER VIEW LED HAL would translate a user viewable state to a user readable LED observation. This could also include Led Flashes as part of the LED display, which isn't currently possible.
Currently the LEDs are written from many different classes, and IMHO used for debugging that class, and need to be configured by the user for the class they want. The LEDs are written from different classes directly and are complex to debug.
A User LED HAL would represent a state that could be requested by the current class. eg System Working ( fast GREEN flashes), or connecting to the cloud (slow GREEN/RED flash), or insufficient battery power to do anything (one slight RED flash)
The MS file system capability could be abstracted at no performance penalty. AS of MS 0.2.0 Its fixed to a specific SD/SPI hardware functionality.
I'm not saying these hardware associations can't be done by modifying current code - but they are painful intertwined for forked implementations to modify and still maintain Arduino style easy code readability.
For instance moving from Mayfly to Adafruit Feather M0 adalogger - the standard Serial is not used, instead Serial1 is the "SerialTty" and there are two other SerialX available.
Using the Adafruit Feather M0 Express there is a second fixed SPI file system with an option to plugin a standard uSD/SPI, which also adds a persistent backed up RTC based on PCF8523 - all at a low cost.
Using the Sodaq Autonomo it implements two physical files systems - a fixed flash SPI and uSD on the same physical SPI.
Serial (COMPLETE see below) - the basic serializer class output can vary between platforms, and is encoded in a number of different .h to a specific piece of hardware.
Assigning all user output to a STANDARD_SERIAL_OUTPUT or DEBUGGING_SERIAL_OUTPUT would be an abstraction - it meets the goal of both enhancing readability and also a HAL assigning it to a specific port - whether that be Serial, Serial1 or SerialUsb or some other eg SerialBlueTooth.
Similarly SerialModem or SerialSDI12 would be a clear readable documentation for a physically defined platform interface, that is mapped to different physical ports on different platforms even when using the same processor.
I'd welcome other discussions of what might work to abstract.
I think there's quite a bit more abstraction that could be done. Several major project do this - think Blynk or Arudino JSON.
I'm surprised you picked on Serial, though. Within the library itself, that's already dealt with via the MS_DBG and PRINTOUT functions. The serial calls are only used in the example programs.
Hey I'm just trying to point out what is PAINFUL when setting up new projects and the real world, not just "examples".
I'm assuming the goal is for easy to maintain code in reliable live systems.
Often HAL is about the basics for people that follow and need to find hard to see bugs, or tweak it to a new platform.
So maybe I'm missing something ..... can I change one location to switch all output to the user (TTY) to go to say Serial1 or SerialUSB and easily do that for the carefully sprinkled debug environment from the standard platformio.ini setup? (so that I can easily do this when building with release mode, and not have to go to a complex developer debug mode.
It seems to me: For (example) menu_a_lca_carte.ino standard output is hardcoded to Serial. and then each MS class/subsystem .h has to be individually edited and assigned If I want to turn on LoggerBase debug which is MS_DBG it seems like it is defined in ModSensorsDebugger.h as STANDARD_SERIAL If I want to turn on LoggerModem debug which is MS_MOD_DBG which seems like it is defined loggerModem.h:MODEM_DEBUGGING_SERIAL_OUTPUT and I have to edit the .h and define MODEM_DEBUGGING_SERIAL_OUTPUT So for each module I need to edit the .h file and then correctly send it to the right serial for that platform ....
Then there is the problem of how to edit the files in ModularSensors/src v ModularSenspors/examples/menu_a_la_cart and what does platformIO actually build - not to mention chaotic merge issues when forked.
Ideally there is a define in one place for all serials - eg SerialTTY output for terminal/debug.
Then each modules debug can be turned in the platform.ini when required with
build_flags = -DLoggerModem_DBG
or more generically
build_flags = -D
So right now I'm spinning my wheels digging through code to find out why 0.19.6/Mayfly is occasionally - every few hours to a day - not talking to the Xbee S6. It has been running for a week on earlier versions with good stability - though as discussed elsewhere it does occasionally even fail after running for some weeks. I suspect its related to the changes for the periodic time synchronization - but I haven't got to prove it yet.
As code stabilizes - which is fantastic to get there - being able to easily turn the debug helps flush out the harder to find bugs.
Hence my feedback that it would be nice to make the serials more extensible /HAL.
I really appreciate your great classes and application level functionality but I'm wasting a lot of time in keeping reliability in the white water of the wake.
I think you're pain is related to the modifications you've made. I barely change anything to switch between boards, just which env I'm using in the platformio file. And I switch between programs with just the src_dir setting.
I can change all the example programs to use the "PRINTOUT()" macro instead of Serial.print(). Then you could change the output of everything from a printout statement by adding the build flag -DSTANDARD_SERIAL_OUTPUT=Serial.
I can add a higher-level macro so that the specific endpoint the debug print out goes to is easier to set universally and then you could only set it once before including ModSensorDebugger.
I find it to be much more painful to turn on debugging via a global build flag. Anytime you change a global build flag in your platformio.ini file every single library has to be rebuilt. Every. Single. One. Every single time you change the global build flag. If you only modify 1 file, like the header for the sensor you're trying to debug, platformio is generally very good about only detecting changed files and only rebuilding those. You may have much faster computer than me, but my computer takes two minutes or so to rebuild everything and I have no interest in waiting that long. The debugging printouts are also not intended to be used in production. They're really for code debugging, not for sitting and watching your logger work.
The MS_MOD_DBG vs MS_DBG was historical, related to the way TinyGSM was implemented as a header-file-only library with no parent class. That's been changed (TinyGSM now has a modem parent class) so now MS_MOD_DBG is gone and only MS_DBG exists.
I'm guessing the problem with the Bee is related to the fact that the Mayfly only implements the CTS pin, not the XBee's true status pin. Sometimes the XBee goofs up and holds CTS low even when it's asleep. ModularSensors sees that that pin is low and assumes the board must be awake and doesn't try to re-wake it. I think if we'd checked the real XBee sleep/on pin (13, not 12) we'd know better. I'm still trying to come up with a work-around, but I don't have anything really working yet. I'm nearly ready to solder in another jumper wire just so I can see what's going on.
Also, have you tried a git gui like GitKraken for messy merges. It's very helpful.
Hey I'm a fan of ModularSensors .. and GIT by LinusT is the greatest SCM, I like visual MergeMeld .. GitKraken looks interesting though planning merges and keeping traceable stability is more of the challenge than doing it. PlatformIO (I'm using VisualStudio) is great, and it finds conflicts in merges with suggestions.
Platformio also implements a make like build -though the distributed ModularSensors is complicated to do debug and release environments because of all the distributed library.json - I'm still learning it - way better than the old ArduinoIDE. When it comes to building on Win10 it seems something else part of "Windows Explorer" istaking cpu cycles. The Antivirus (mine is bitdefender) can also get suspicous of tools.
However a big time waster is when I do a refresh of .piolibs, which takes me(my network) 15min++ and I've yet again on 0.19.6 got bitten by "Error: Could not find ADS1X15 dependency for EnviroDIY_ModularSensors library". The fix was never testable on 0.19.6
Ugh!! - so back into the white water of merging to 0.21.2 - so I anticipate a 4hr + hit to get a traceable working environment back and start testing stability again.
I designed my first whole board hw and fw on an Intel 8048 in 1982, (its 'orrible uP don't even look it up) and have worked with embedded processors ever since, sometimes with embedded teams of up to 30people with a variety of SCM ... and its AMAZING to use distributed development with ModularSensors over git .
Whats great about ModularSensors and components is its accessibility over git - but accurate caching is key to productivity. Stability is KEY to ANY computer system - and especially embedded system - which are expected to work unattended for years.
Large companies have gone under when they've lost their stability and architecture of their systems.
I implement stability in embedded systems by forcing a reboot if anything undefined happens. That way its pretty invisible to a remote user. I have seen transport systems like "cable network transport" reboot at 3am every morning.
so I'm just floating my perceptions of HAL (from surfing the white water wake) and easily enabling debug are useful. I'm not at all suggesting enabling all debug at one time. I'm sure you know your code backwards... and I may take some time to get into it, - but as I've learnt with teams - sometimes I need to socialize my perceptions of the challenges.
So my HAL on Serial. was about trying to have one place where the logical to physical is defined, instead of hours navigating the code.
PRINTOUT() seems like /*this is a comment that should be meaningful */
but what ever works - seems like UserOutput() or SerialTty() or SerialUser() would be better
I did find for platformio.ini -buried deep in macros in .h.cpp files
-DTINY_GSM_DEBUG=Serial
-DMODEM_DEBUGGING_SERIAL_OUTPUT=Serial
and a few others
and part of my suggestion was a self documenting naming convention build_flags = -Dmodule_name_DEBUG
but also I had other HAL thoughts on SD phy that I'm seeing are getting distributed in 0.21.2
I really appreciate ModularSensor and just trying to add some (i hope) value feedback :)
Onward with merge to 0.21.2.....
I think you need to move more of your modifications into separate files and classes so you don't have these painful merges.
@aufdenkampe are you having problems like multi-hour merge nightmares and debugging complications?
Well thanks for the suggestion .... I actually have my .ino in a separate file - and for stability and discussion purposes I'm tracking menu_a_la_carte.ino.
Oh. Yes, I definitely have a separate ino that I always use. Atom has a fabulous extension called split-diff that makes it really, really easy to compare two files and sync the parts you want. VSCode as a sort-of similar difference viewer, but in the VSCode version, you can't select a change and transfer it between the files making it dramatically less useful.
I don't know anything about HAL... but I did just get this same error message:
Error: Could not find ADS1X15 dependency for EnviroDIY_ModularSensors library
when compiling simple_logging to a completely fresh directory.
Oddly, it worked fine yesterday...
I noticed the ADS1X15 in a recent bug fix, so I might cross-post.
I'm back with a working sandbox on 0.21.2.
Just to be clear my proposal for the User Terminal/Debug interface HAL - what I call SerialTty is to make it easier for users to understand and easily pickup on the program flow in ModularSensors (so that they can enhance what a great package it is)
For me the current system required of developers, often requires editing in ModularSensors/src/.h which causes all sort of knock on problems with Platformio promiscuous/indescriminante build environment that I don't follow and I often end up having errors between different versions of the same.h and so for specific files I've edited in ModularSensors/src/.h I have to delete the copies in ModularSensors\aTree\myExample.piolibdeps\EnviroDIY_ModularSensors\src so it will build.
Then I end up having to flush/delete
ModularSensors\aTree\myExample.piolibdeps\EnviroDIY_ModularSensors
which is painful as it has to go out and pull in a new version of EnviroDIY_ModularSensors - which is close to 400M of database - most of it overhead.
Typically all I'm trying to do is get more insight into the way a subsystem/class is working across different platforms.
so thats my pain. Ahhhh. :) really its painful.!!
The solution I've implemented in my sandbox is to make modifications in
ModularSensors\aTree\myExample\platformio.ini
- and rebuild just the local environment.
For those who have an easy develop setup, the old way still works.
so in the local env platformio.ini
;****build_flags options ;-DSerialTty=Serial ; Assign serial Channel ;-DSTREAMDEBUGGER_DBG ;-DdataPublisherBase_DBG ;-DLoggerBase_DBG ;-DLoggerModem_DBG ;-DSensorBase_DBG ;-DVariableArray_DBG ;-DVariableArray_DeepDBG ;-DVariableBase_DBG ;-DProcessorStats_DBG ;-DExternalVoltage_DBG
each class .h file then has
// FOR DEBUGGING // #define DEBUGGING_SERIAL_OUTPUT Serial #ifdef LoggerModem_DBG #define DEBUGGING_SERIAL_OUTPUT SerialTty #endif //LoggerModem_DBG
Of course I get a delayed "pain" when I come to merge to the next level, as there might be conflicts in the merge ... but have to sort that out then.
ok, changed all so now merely defining the printout/debug/deep_debug output destination will not cause output.
The destination for printout can be set universally with #define STANDARD_SERIAL_OUTPUT and will be set to Serial or SerialUSB if not otherwise defined
The destination for debugging of all kind can be set universally with #define DEBUGGING_SERIAL_OUTPUT and will be set to Serial or SerialUSB if not otherwise defined.
To turn on debugging by modifying individual h files (my personal preference), un-comment the line // #define MS_XXX_DEBUG and/or // #define MS_XXX_DEBUG_DEEP within the h file.
Individual modules can also have their debugging turned on via a PlatformIO ini file (Neil H style) by defining the global build flag -DMS_XXX_DEBUG or -DMS_XXX_DEBUG_DEEP (ie, -DMS_KELLER_DEBUG)
It is no longer (easily) possible to do something strange like send the debugging printouts from the Logger Serial2 and the debugging printouts from the VariableArray to Serial3. That was not something I'd ever intended to support, it just used to be easier to do. I can't imagine why you'd really want to do something like this, but if you miss it, let me know.
Hey thanks @SRGDamia1
your a gem. I think it will really help the next wave of scientists/programmers who want to dig through and do their own tweaking.
Looking through
https://github.com/EnviroDIY/ModularSensors/commit/fc2f25f2857e73c7ef84436bbb73c2691f9e02ca
just a slight note - I was suggesting the XXX follow the filename so that it doesn't need decoding. Though still looks great to me. :)
I've just added this to the first section - the reason is that I'm working through trying to make one Autonomo greenLED represent a function (it only has one LED). So this is my experience of that process - I probably have missed something, but still trying to find it.
If I do a special Autonomo Green blinkey() module then I can control the LED.
However with ModularSensors (at least in my fork,) something happens and its not easy to figure out what - but the LED is not controllable - possibly a number of modules toggling the bit. Still figuring it out.
So I've added Hardware Abstration Layer of LEDs for discussion.
On the one hand Blinkey() is a great teaching tool ~~ however ~~
IHMO as an electronic designer/engineer - the transition from a Blinkey() to a meaning of the LED is part of the system design and benefits from being clearly defined.
For a working system that is going to be deployed into the field, sometimes under challenging conditions, the deployment "technician", who is juggling many other issues (often in very adverse hot or cold environmental conditions or a local partner viewing) - needs to know that the electronics - ModularSensors - is working at the end of the deployment. Typically the final instructions need to be - press RESET, monitor LEDs and when X happens (say 1 fast red flash) its connecting to the sensors (yeah) - and when Y (say green/red slow flashes) its connecting to the cloud (whew!!).
If there is an API for the User Feedback (ie LEDs but could be something else) - any tutorials can make it easier to vary the visual/user feedback and discuss what is needed at the field site to declare a complete installation.
(Added above 2019Mar26) The board LED(s) represent a user view of the state of the system - is it working getting data (fast RED flashes), or connecting to the cloud (slow GREEN/RED flash), or not enough power to do anything (one slight GREEN flash) A HAL would translate a state to a user readable LED observation. Currently the LEDs are written from many different classes, and its confusing unless you know the code as to what do the LED colours/flashes mean. Also since the LEDs are written from different classes directly to the hardware, its hard to debug.
Just been merging to 0.21.4 thanks @SRGDamia1 for the aligned debugging names https://github.com/EnviroDIY/ModularSensors/commit/ad2108020c2b8cb47f57d8828467e41b8b3849d5
The system is pretty mature to contemplate any changes like this so closing