espressif
ESP-AWS-IoT support for fleet provisioning (CA-182)
Hello please when the ESP-AWS-IoT component will support fleet provisioning ?
You can achieve Fleet Provisioning right now. I did it both in the older version and in this new beta version. Everything is just a matter of subscription and publishing. There is quite a bit to set up over at AWS, but the instructions there are correct and complete. Order of operation is important. And you may have one sticking point when you receive the new certs. AWS may possibly send the certs to you with 2 character line feeds. You'll need to strip those out and insert one character line feeds at the correct 64 chars positions. Other than this --- the provisioning does work fairly well.
Hello @SolidStateLEDLighting , could you explain a little more how to carry out the provisioning of fleets with OTA functionality of the release/beta version?
All of the IOT services are not much more than subscriptions, publishing, and the handling of response messages.
If you get your project organized correctly, you can do a client log-in (no Thing identity), a Fleet Provisioning, a Shadow connection (Thing identity), a Job service, and OTA -- with just pub/sub and messages.
The big problem with the supplied OTA project is that it is not made for integration with all the other services. I deconstructed their process and merged it with all my other work.
I'm not using HTTPs for downloading binaries right now -- I do everything with MQTT. MQTT is mandatory for control messages, but you can stream with it for the OTA binary downloads too. So, I find it easier to just MQTT_ProcessLoop() for everything in one location.
The OTA Agent software example provided is written to be POSIX universal for a hand-full of hardware platforms. So, it is very unfriendly to ESP32 directly. And this guy (or girl) abstracts the work with function pointers (which drives me bananas), because you can't follow the code in a progressive and logical way. I like project organization that teaches you what to do (or reminds you what you did).
@SolidStateLEDLighting Thanks for your feedback. I think I now have a clear understanding of the process. Reviewing the AWS documentation suggests the following workflow.
This is a fleet provisioning by claim ( JITP )
I will be working on conditioning the OTA example for fleet provisioning and I would appreciate if you know any (documentation or examples) that can support me.
And another question that does not let me sleep -_- Is it possible to update the code signature with which the firmware file to be updated is signed via OTA, since this signature is in flash and has an expiration date in AWS?
I think that Fleet Provision by Claim and JITP are slightly different things.
The OTA example has nothing to do with provisioning. It is one of the reasons that I don't like it. It does not get along with all other services.
I'm not completely up to speed on the code signing for OTA. I make a custom OTA job and supply the cert in the window provided (third selection). That cert is sent by AWS in the Job description. This allows me for now to download a non-encrypted binary (AWS doesn't know that its unsigned). You wouldn't be able to see any of this inside the OTA tool that they provide because all that is hidden behind the scenes. I don't even know if you have the option of using my approach. That OTA example is not documented well enough inside the code to know all that you need. Externally -- no documentation at all. As I mentioned in previous email -- I tore it apart and followed its pattern to achieve the similar results.
I believe that updating the code signature is something that can be achieved inside AWS Certificate Manager -- but I'm not using that service for OTA right now.
From: EmbeddedDBL @.> Sent: Wednesday, January 19, 2022 7:19 AM To: espressif/esp-aws-iot @.> Cc: keith ssledlighting.com @.>; Mention @.> Subject: Re: [espressif/esp-aws-iot] ESP-AWS-IoT support for fleet provisioning (CA-182) (Issue #87)
@SolidStateLEDLightinghttps://github.com/SolidStateLEDLighting Thanks for your feedback. I think I now have a clear understanding of the process. Reviewing the AWS documentation suggests the following workflow.
This is a fleet provisioning by claim ( JITP ) [image]https://user-images.githubusercontent.com/94095787/150033039-acdab31c-ff33-4c21-b8ec-75320cc49037.png
I will be working on conditioning the OTA example for fleet provisioning and I would appreciate if you know any (documentation or examples) that can support me.
And another question that does not let me sleep -_- Is it possible to update the code signature with which the firmware file to be updated is signed via OTA, since this signature is in flash and has an expiration date in AWS?
— Reply to this email directly, view it on GitHubhttps://github.com/espressif/esp-aws-iot/issues/87#issuecomment-1015921549, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AGGOKE2XMTTTJUNHDFXIRYLUWXYR5ANCNFSM5LVUHX5A. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you were mentioned.Message ID: @.***>
@SolidStateLEDLighting Hello again. I appreciate your comments.
Ask you if you used these AWS APIs for fleet provisioning by claim ? https://docs.aws.amazon.com/es_es/iot/latest/developerguide/fleet-provision-api.html#create-cert-csr-request-payload
This method called CSR is the same as by claim?
Yes, I did fleet provisioning by claim. I believe it is somewhat different than the process which uses the Certificate Signing Request. I did not read up on the CSR method very much.
I did not use APIs. There is no programming interface other than subscribing and publishing to topics which apply to the provisioning process (by claim).
You need to set up the AWS side with a provisioning template -- and all the correct roles and policies. That takes careful attention to detail. The AWS documents that explain all this are correct.
Keith
From: EmbeddedDBL @.> Sent: Wednesday, January 26, 2022 8:34 AM To: espressif/esp-aws-iot @.> Cc: keith ssledlighting.com @.>; Mention @.> Subject: Re: [espressif/esp-aws-iot] ESP-AWS-IoT support for fleet provisioning (CA-182) (Issue #87)
@SolidStateLEDLightinghttps://github.com/SolidStateLEDLighting Hello again. I appreciate your comments.
Ask you if you used these AWS APIs for fleet provisioning by claim ? https://docs.aws.amazon.com/es_es/iot/latest/developerguide/fleet-provision-api.html#create-cert-csr-request-payload
This method called CSR is the same as by claim?
— Reply to this email directly, view it on GitHubhttps://github.com/espressif/esp-aws-iot/issues/87#issuecomment-1021739911, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AGGOKE77Q3UUUXGLEMQBI2LUX46RZANCNFSM5LVUHX5A. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you were mentioned.Message ID: @.***>
Thanks for your comment. @SolidStateLEDLighting I now understand better what needs to be done, I think I am following the steps correctly but even so I am not getting a response from AWS.
1 Connect to MQTT with claim certificate 2 I subscribe to the answer topics $aws/certificates/create/payload-format/accepted , ....rejected 3 Post a message to the topic $aws/certificates/create/payload-format This is where the question arises what message I have to publish since it mentions that it is an empty payload-format.
I'm not sure if the problem is the message you sent when making the request. Thank you again for your comments. Now try the CreateCertificateFromCsr method.
Notes:
- I have no idea how the provisioning with CSR works. I didn't do it this way so I can't help you in this method. I did it "by claim"
- Yes -- read the docs carefully on all the topics needed for your specific action
- {} <-- (this is the empty payload, but literally nothing may also work) I experimented and read documents to figure this out.
From: EmbeddedDBL @.> Sent: Wednesday, February 2, 2022 7:16 AM To: espressif/esp-aws-iot @.> Cc: keith ssledlighting.com @.>; Mention @.> Subject: Re: [espressif/esp-aws-iot] ESP-AWS-IoT support for fleet provisioning (CA-182) (Issue #87)
Thanks for your comment. @SolidStateLEDLightinghttps://github.com/SolidStateLEDLighting I now understand better what needs to be done, I think I am following the steps correctly but even so I am not getting a response from AWS.
1 Connect to MQTT with claim certificate 2 I subscribe to the answer topics $aws/certificates/create/payload-format/accepted , ....rejected 3 Post a message to the topic $aws/certificates/create/payload-format This is where the question arises what message I have to publish since it mentions that it is an empty payload-format.
I'm not sure if the problem is the message you sent when making the request. Thank you again for your comments. Now try the CreateCertificateFromCsr method.
— Reply to this email directly, view it on GitHubhttps://github.com/espressif/esp-aws-iot/issues/87#issuecomment-1027381912, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AGGOKE2Q5E7W65K7UJFI2FDUZBSV5ANCNFSM5LVUHX5A. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you were mentioned.Message ID: @.***>
Hello Try to subscribe to the answer topic $aws/certificates/create/json/accepted and publish an empty msg to $aws/certificates/create/json
Why would I want to follow this suggestion? For what purpose?
From: helaagardabbou @.> Sent: Wednesday, February 2, 2022 4:51 PM To: espressif/esp-aws-iot @.> Cc: keith ssledlighting.com @.>; Mention @.> Subject: Re: [espressif/esp-aws-iot] ESP-AWS-IoT support for fleet provisioning (CA-182) (Issue #87)
Hello Try to subscribe to the answer topics $aws/certificates/create/json/accepted and publish an empty msg to $aws/certificates/create/json
— Reply to this email directly, view it on GitHubhttps://github.com/espressif/esp-aws-iot/issues/87#issuecomment-1027711022, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AGGOKE45S6KIBTSUJKB3TS3UZDWCNANCNFSM5LVUHX5A. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you were mentioned.Message ID: @.***>
because payload format must be cbor or json $aws/certificates/create/cbor or $aws/certificates/create/json
I have had success with Fleet Provisioning by Claim.
Why are you providing me this suggestion?
From: helaagardabbou @.> Sent: Wednesday, February 2, 2022 4:55 PM To: espressif/esp-aws-iot @.> Cc: keith ssledlighting.com @.>; Mention @.> Subject: Re: [espressif/esp-aws-iot] ESP-AWS-IoT support for fleet provisioning (CA-182) (Issue #87)
because payload format musht be cbor or json
— Reply to this email directly, view it on GitHubhttps://github.com/espressif/esp-aws-iot/issues/87#issuecomment-1027714041, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AGGOKE6MW5U6VEUHILP6NOLUZDWQTANCNFSM5LVUHX5A. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you were mentioned.Message ID: @.***>
This suggestion is a response for empty payload-format error of EmbeddedDBL
Of course.... sorry for the confusion. I'm just looking at this in my email utility.
From: helaagardabbou @.> Sent: Wednesday, February 2, 2022 4:58 PM To: espressif/esp-aws-iot @.> Cc: keith ssledlighting.com @.>; Mention @.> Subject: Re: [espressif/esp-aws-iot] ESP-AWS-IoT support for fleet provisioning (CA-182) (Issue #87)
This suggestion is a response for empty payload-format error of EmbeddedDBL
— Reply to this email directly, view it on GitHubhttps://github.com/espressif/esp-aws-iot/issues/87#issuecomment-1027716456, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AGGOKE6PSLPNIQWOZ7RWPKLUZDW3LANCNFSM5LVUHX5A. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you were mentioned.Message ID: @.***>
@helaagardabbou That's right, first I subscribe to the answer topics $aws/certificates/create/cbor/accepted and rejected
Then I publish the empty message in cbor $aws/certificates/create/cbor format but I do not get a response from the server, however when I send a badly formatted message I do get a response in the rejected response topic and this has me confused.
@EmbeddedDBL the publish and the subscribe are done with the same client ? that's mean You did subscribe in the device and the publish action with a console client?
Please don't use cbor (compact binary object representation) until you need to stream data. cbor can be used, but you have encode it with a cbor encoder. The results you get have to be decoded before inspection.
Use json (javascript object notation) so you can easily read the message in when they come back to you.
@SolidStateLEDLighting I totally agree with you also u can easily know the error with json format
@helaagardabbou If it is the same client with which I establish an MQTT connection, when I publish the request message in the IoT security/Certificates console I can see that a new certificate is created but I do not receive anything in the /accepted topic.
Ok I appreciate your comments I will change to Json
@EmbeddedDBL u must publish mqtt message with the device which is subscribed on the /accepted topic
Everything is done in the same MQTT connection if that connection is authorized (policy/roles) are defined correctly at AWS. If the cert/key are being created, that you are at the first step. Make double sure you are subscribing correctly to the right topics. Cloud watch can assist you in seeing the actions and there may be some help in seeing errors there (like what topics you have subscribed to, etc).
K.
From: EmbeddedDBL @.> Sent: Thursday, February 3, 2022 12:32 AM To: espressif/esp-aws-iot @.> Cc: keith ssledlighting.com @.>; Mention @.> Subject: Re: [espressif/esp-aws-iot] ESP-AWS-IoT support for fleet provisioning (CA-182) (Issue #87)
@helaagardabbouhttps://github.com/helaagardabbou If it is the same client with which I establish an MQTT connection, when I publish the request message in the IoT security/Certificates console I can see that a new certificate is created but I do not receive anything in the /accepted topic.
Ok I appreciate your comments I will change to Json
— Reply to this email directly, view it on GitHubhttps://github.com/espressif/esp-aws-iot/issues/87#issuecomment-1028123426, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AGGOKE42AGGRG7QDWIYC4DLUZFMBNANCNFSM5LVUHX5A. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you were mentioned.Message ID: @.***>
I publish this in the topic $aws/certificates/create/json in JSON format char payloadBuffer2[] = " "; with the intention that it fails to see if I choose an answer in the error topic.
When I post a correct message I get no response in my mqtt callback function char payloadBuffer2[] = "{"certificateSigningRequest":" " }";
Here is what the topics look like for Fleet Provisioning by Claim...
// First Step std::string create_topic = "$aws/certificates/create/json"; // This is a publish topic std::string create_json_accepted_topic = "$aws/certificates/create/json/accepted"; // This is a subscribe topic std::string create_json_rejected_topic = "$aws/certificates/create/json/rejected"; // This is a subscribe topic
// Second Step std::string provision_json_topic = "$aws/provisioning-templates/Provisioning_Template_V2/provision/json"; // This is a publish topic std::string provision_json_accepted_topic = "$aws/provisioning-templates/Provisioning_Template_V2/provision/json/accepted"; // This is a subscribe topic std::string provision_json_rejected_topic = "$aws/provisioning-templates/Provisioning_Template_V2/provision/json/rejected"; // This is a subscribe topic
Notice that I must provide a template which is accessible to my provisioning process.
From: EmbeddedDBL @.> Sent: Thursday, February 3, 2022 2:52 AM To: espressif/esp-aws-iot @.> Cc: keith ssledlighting.com @.>; Mention @.> Subject: Re: [espressif/esp-aws-iot] ESP-AWS-IoT support for fleet provisioning (CA-182) (Issue #87)
I publish this in the topic $aws/certificates/create/json in JSON format char payloadBuffer2[] = " "; with the intention that it fails to see if I choose an answer in the error topic.
When I post a correct message I get no response in my mqtt callback function char payloadBuffer2[] = "{"certificateSigningRequest":" " }"; [image]https://user-images.githubusercontent.com/94095787/152218487-ee692567-dbda-4907-8568-e94f57a47ced.png
— Reply to this email directly, view it on GitHubhttps://github.com/espressif/esp-aws-iot/issues/87#issuecomment-1028254974, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AGGOKE7MLN6J6PVG6KJRGX3UZF4QPANCNFSM5LVUHX5A. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you were mentioned.Message ID: @.***>
@SolidStateLEDLighting Upon completion of step one before proceeding to step two, the response from the "$aws/certificates/create/json/accepted" topic is received in a callback registered in MQTT_Init( );
Here my problem arises I do not receive any response I do not know if it is my implementation or my AWS configuration
You'll need to set up Cloud watch to confirm that the AWS system created a Publish Out message. That message will be your payload response.
Instructions on how to get Cloud watch to show you all the IOT traffic is documented. Just follow the the documents.
Everything has to be perfect for everything to work. Order of operation is critical. Having your topics correct is critical.
From: EmbeddedDBL @.> Sent: Thursday, February 3, 2022 8:48 AM To: espressif/esp-aws-iot @.> Cc: keith ssledlighting.com @.>; Mention @.> Subject: Re: [espressif/esp-aws-iot] ESP-AWS-IoT support for fleet provisioning (CA-182) (Issue #87)
@SolidStateLEDLightinghttps://github.com/SolidStateLEDLighting Upon completion of step one before proceeding to step two, the response from the "$aws/certificates/create/json/accepted" topic is received in a callback registered in MQTT_Init( );
Here my problem arises I do not receive any response I do not know if it is my implementation or my AWS configuration
— Reply to this email directly, view it on GitHubhttps://github.com/espressif/esp-aws-iot/issues/87#issuecomment-1028496772, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AGGOKE6CZZE2TKH5R54KDOTUZHGELANCNFSM5LVUHX5A. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you were mentioned.Message ID: @.***>
Here is another serious possibility....
The demos are not well written (but they are simple enough). They block in the MQTT_ProcessLoop call for a small period.
The time it takes for you to receive a subscription or publish is small.
The time it takes for you to receive a payload is about 40 times longer in some cases. So, you'll need to extend out your time in MQTT_ProcessLoop long enough to be sure you receive the payload.
I have written my project here to never leave MQTT_ProcessLoop area unless I have something to send. Not leaving this loop is important because you don't know when the system is sending the MQTT keep alive pings -- and your code must process them behind the scenes for you in this same loop.
K.
From: keith ssledlighting.com @.> Sent: Thursday, February 3, 2022 8:54 AM To: espressif/esp-aws-iot @.> Subject: Re: [espressif/esp-aws-iot] ESP-AWS-IoT support for fleet provisioning (CA-182) (Issue #87)
You'll need to set up Cloud watch to confirm that the AWS system created a Publish Out message. That message will be your payload response.
Instructions on how to get Cloud watch to show you all the IOT traffic is documented. Just follow the the documents.
Everything has to be perfect for everything to work. Order of operation is critical. Having your topics correct is critical.
From: EmbeddedDBL @.> Sent: Thursday, February 3, 2022 8:48 AM To: espressif/esp-aws-iot @.> Cc: keith ssledlighting.com @.>; Mention @.> Subject: Re: [espressif/esp-aws-iot] ESP-AWS-IoT support for fleet provisioning (CA-182) (Issue #87)
@SolidStateLEDLightinghttps://github.com/SolidStateLEDLighting Upon completion of step one before proceeding to step two, the response from the "$aws/certificates/create/json/accepted" topic is received in a callback registered in MQTT_Init( );
Here my problem arises I do not receive any response I do not know if it is my implementation or my AWS configuration
— Reply to this email directly, view it on GitHubhttps://github.com/espressif/esp-aws-iot/issues/87#issuecomment-1028496772, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AGGOKE6CZZE2TKH5R54KDOTUZHGELANCNFSM5LVUHX5A. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you were mentioned.Message ID: @.***>
I already achieved provisioning by claim using CreateCertificateFromCsr it required more work but it works great. I couldn't get a response with CreateKeysAndCertificate so I opted for CreateCertificateFromCsr. Just thank you for your comments @SolidStateLEDLighting and @helaagardabbou 👍