leshan
leshan copied to clipboard
eclipse-leshan
Question - How we check compliance Leshan with standard
How are we sure that Leshan project is properly implemented with standards? Do you execute some periodically external tests or compliance is checked only while code reviewing?
How are we sure that Leshan project is properly implemented with standards?
When you talk about "standards", you mean LWM2M or any standard that Leshan is based on ?
Do you execute some periodically external tests or compliance is checked only while code reviewing?
I don't know about this kind of external tests. Currently we rely on code review + our unit tests + feedback of our community.
Sorry, I didn't crystallize what I mean :smile: Of course I mean LWM2M :smile:
Currently we rely on code review + our unit tests + feedback of our community.
Ok, we currently study of possibility for compliance testing with a third party. But only once because it costs :smile:
Ok, we currently study of possibility for compliance testing with a third party.
That could be great. :+1: Could you share who does this kind of compliance test ?
But only once because it costs smile
If this is a one shot maybe we should keep it for a the final RC release, or ?
For some reasons we have to perform tests sooner rather than later...
Nevertheless, maybe you have some preferences regarding tests? The potential options that we can perform are:
int-0 Client Initiated Bootstrap
int-1 Client Initiated Bootstrap Full (PSK)
int-2 Client Initiated Bootstrap Full (Cert)
int-3 Simple Bootstrap from Smartcard
int-4 Bootstrap Delete
int-5 Server Initiated Bootstrap
int-6 Bootstrap Sequence
int-7 Fallback to bootstrap
int-8 Bootstrap Read
int-9 Bootstrap and Configuration Consistency
int-101 Initial Registration
int-102 Registration Update
int-103 Deregistration
int-104 Registration Update Trigger
int-105 Discarded Register Update
int-106 TCP Binding
int-107 Extending the lifetime of a registration
int-108 Turn on Queue Mode
int-109 Behavior in Queue Mode
int-201 Querying basic information in Plain Text format
int-203 Querying basic information in TLV format
int-204 Querying basic information in JSON format
int-205 Setting basic information in Plain Text format
int-211 Querying basic information in CBOR format
int-212 Setting basic information in CBOR format
int-215 Setting basic information in TLV format
int-220 Setting basic information in JSON format
int-221 Attempt to perform operations on Security Object (ID 0)
int-222 Read on Object
int-223 Read on Object Instance
int-224 Read on Resource
int-225 Read on Resource Instance
int-226 Write (Partial Update) on Object Instance
int-227 Write (replace) on Resource
int-228 Write on Resource Instance
int-229 Read-Composite Operation
int-230 Write-Composite Operation
int-231 Querying basic information in SenML JSON format
int-232 Querying basic information in SenML CBOR format
int-233 Setting basic information in SenML CBOR format
int-234 Setting basic information in SenML JSON format
int-235 Read-Composite Operation on root path
int-236 Read-Composite Partial Presence
int-237 Read on Object without specifying Content-Type
int-241 ExecuResource Rebooting the device
int-256 Write Operation Failure
int-257 Write-Composite Operation
int-260 Discover Command
int-261 Write-Attribute Operation on a multiple resource
int-280 Successful Read-Composite Operation
int-281 Partially Successful Read-Composite Operation
int-301 Observation and Notification of parameter values
int-302 Cancel Observations using Reset Operation
int-303 Cancel observations using Observe with Cancel parameter
int-304 Observe-Composite Operation
int-305 Cancel Observation-Composite Operation .
int-306 Send Operation
int-307 Muting Send
int-308 Observe-Composite and Creating Object Instance
int-309 Observe-Composite and Deleting Object Instance
int-310 Observe-Composite and modification of parameter values
int-311 Send command
int-401 UDP Channel Security Pre-shared Key Mode
int-402 UDP Channel Security Certificate Mode .
int-403 UDP Channel Security Certificate Mode (Server Identify Verification Failure) .
int-404 TCP Channel Security Certificate Mode ..
int-405 OSCORE Security
int-651 Check Access to the Resources
int-652 Querying the firmware version from the client
int-680 Create Object Instance
int-685 Delete Object Instance
int-701 Querying the readable resources of object
int-710 Basic Observation and notification on Connectivity Monitoring Object Resources
int-751 Querying the readable resources
int-755 Setting the wriResource Package
int-756 Setting the wriResource Package URI
int-760 Basic Observation and notification on Firmware Update Object Resources
int-770 Successful Firmware update (via COAP)
int-771 Successful Firmware update (via alternative mechanism)
int-772 Error Case 1 firmware installation without downloaded package
int-773 Error Case 2 shortage of storage memory
int-774 Error Case 3 out of memory
int-775 Error Case 4 Connection lost during download (package URI)
int-776 Error Case 5 Package Integrity check failure
int-777 Error Case 6 unsupported package type
int-778 Error Case 7 invalid URI (package URI)
int-779 Error Case 8 Unsuccessful Firmware Update
int-801 Querying the readable resources of object
int-810 Basic Observation and notification on Location Object Instance
int-901 Querying a Data Collection from Connectivity Object Instance
int-905 Setting the wriresources
int-910 Basic Observation and notification on Connectivity Monitoring Object Instance
int-950 Multi-Servers Registration
int-951 Multi-Servers & Attributes
int-1200 Querying the readable resources of Object ID:10
int-1201 Querying the readable resources of Object ID:10 in version 1.1
int-1202 Setting the Power Saving Mode Resources of Object ID:10 (version 1.1 only)
int-1203 Observation and notification on Object ID:10 related to Power Saving Mode Resources (version 1.1 only)
int-1204 Observation and notification on Object ID 10 related to timers
int-1250 APN configuration
int-1350 Bearer Selection
int-1630 Create Portfolio Object Instance
int-1635 Delete Portfolio Object Instance
int-1900 Observation and notification on Object ID 19
int-1901 Setting the wriResource Data
int-2000 Control the Log Function
int-2001 Querying the readable resources
What do you think, some tests would be more preferable than other? Or some tests should not be performed because lack of implementation?
I think the tests are based on: http://www.openmobilealliance.org/release/lightweightm2m/ETS/OMA-ETS-LightweightM2M-V1_1-20190912-D.pdf
Tests are for clients, servers or bootstrap servers ?
What do you think, some tests would be more preferable than other?
I don't know :thinking: ... I guess the most we tests the better it is :sweat_smile: ? maybe the cost depends of the number of tests ?
Or some tests should not be performed because lack of implementation?
I can have a first look and try to identify tests which will not pass.
Tests are for clients, servers or bootstrap servers ?
We are focusing on server side. The client code is by the way.
maybe the cost depends of the number of tests ?
That's right. But we don't specified yet how many tests we will perform. We still debate about that.
I can have a first look and try to identify tests which will not pass.
Thx! It will help us a lot :smiley:
A general remarks, I don't really know what you will tests ? I mean demos or library ?
Demos works pretty much out of the box but does not support everything supported by the library. Library supports much more but that means you need to code a runnable program with it to make it works ? Or maybe you test your product which integrates Leshan ?
Good point. Leshan is just a library and without proper usage it can't be tested enough. I think the demos might be not enough to perform all tests. We have to think about that.
| test | comment |
|---|---|
| int-3 Simple Bootstrap from Smartcard | I think this mainly concerns client side but by default we don't really support smartcard even if I'm not sure your what it could means to support it in the library |
| int-106 TCP Binding | not yet implemented, I'm not even sure we have a coap+tcp mature java implementation available |
| int-235 Read-Composite Operation on root path | Read Composite is implemented but not about targeting root path |
| int-260 Discover Command | write attributes not implemented at client side |
| int-261 Write-Attribute Operation on a multiple resource | write attributes not implemented at client side |
| int-301 Observation and Notification of parameter values | write attributes not implemented at client side |
| int-307 Muting Send | mute is not implemented at client side but should be pretty easy |
| int-404 TCP Channel Security Certificate Mode .. | TCP not implemented |
| int-405 OSCORE Security | in development but not manage by me, so not so sure of the current state. Currently not in master |
| from 651 to 910 | this is about testing object implementation but currently client library is more about letting you implement your object |
| int-950 Multi-Servers Registration | multi server at client side is not implemented |
| int-951 Multi-Servers & Attributes | multi server at client side is not implemented |
| from 1200 to 2001 | this is about testing object implementation but currently client library is more about letting you implement your object |
I did a really quick look. The tests not mentioned above :point_up: should be OK
I think this question is maybe a bit related to : https://github.com/eclipse/wakaama/issues/617
(Just some thought about this, maybe a security audit about Leshan would be a better investment rather than a LWM2M specification compliance one)
It seems that most of http://www.openmobilealliance.org/release/lightweightm2m/ETS/OMA-ETS-LightweightM2M-V1_1-20190912-D.pdf pass with Zephir, see : https://github.com/zephyrproject-rtos/zephyr/pull/64013