VEN telemetry status reports do not appear to be supported

[francisrsandoval]

It appears from ReportService.update_report that while a VEN’s client report returns a table of intervals, there is no definition in the platform for the interval payload to express the oadrOnline or oadrManualOverride values as described in the OpenADR 2.0b specification.

[stan-janssen]

Correct, I don't have support for those properties yet. Do you have a clear picture of what is needed to support these? Otherwise I'll look into it myself later. Thanks.

[stan-janssen]

All right, it looks to me like these properties are needed for the TELEMETRY_STATUS report types. The following elements can be provided in for each interval:

• oadrOnline (boolean, required)
• oadrManualOverride (boolean, required)
• oadrLoadControlState (optional)
  • oadrCapacity
    • oadrMin
    • oadrMax
    • oadrCurrent
    • oadrNormal
  • oadrLevelOffset (optional)
    • oadrMin
    • oadrMax
    • oadrCurrent
    • oadrNormal
  • oadrPercentOffset (optional)
    • oadrMin
    • oadrMax
    • oadrCurrent
    • oadrNormal
  • oadrSetPoint (optional)
    • oadrMin
    • oadrMax
    • oadrCurrent
    • oadrNormal

Proposed Design

Perhaps a neat design might be the following.

Allow the data-collection function to return either of these tuples:

• bool, bool: interpreted as oadrOnline, oadrManualOverride (the first may be None if you don't want to include it.
• bool, bool, dict: interpreted as oadrOnline, oadrManualOverride, oadrLoadControlState

Any of the above may also be prefixed with a datetime.datetime value to indicate a different time than datetime.now().

To register your TELEMETRY_STATUS report:

def status_callback():
    return True, False

client.add_report(callback=status_callback, 
                  resource_id='Device001',
                  report_name='TELEMETRY_STATUS')

Or a status report that contains further information on the load control effects:

def status_callback_full():
    return True, False, {'capacity': {'min': 0,
                                      'max': 10,
                                      'current': 5,
                                      'normal': 8}}

client.add_report(callback=status_callback_full, 
                  resource_id='Device002,
                  report_name='TELEMETRY_STATUS')

On the VTN side, the TELEMETRY_STATUS could already be recognized in 'compact' mode by the measurement='Status' parameter, and in full mode by inspecting report_name, so that should already work.

If this seems like a workable solution, I will implement this for the next release.

[stan-janssen]

I implemented a first version of the TELEMETRY_STATUS reporting functionality in a separate branch. The interface is as described above.

It does come with a small change to how the VTN hands off data to the report callback function: it used to supply a list of (datetime.datetime, value) items that corresponded with the measured intervals:

[(datetime.datetime(...), 1.0),
 (datetime.datetime(...), 2.0)]

it now supplies a list of dicts that contain the report data in the following format:

[{'dtstart': datetime.datetime(...), 'value': 1.0},
 {'dtstart': datetime.datetime(...), 'value': 2.0}]

Or, for the status reports:

[{'dtstart': datetime.datetime(...),
  'resource_status': {'online': True,
                      'manual_override': False,
                      'capacity': {'min': 0,
                                   'max': 10,
                                   'current': 5,
                                   'normal': 8}}},
 {'dtstart': datetime.datetime(...),
  'resource_status': {'online': True,
                      'manual_override': False,
                      'capacity': {'min': 0,
                                   'max': 10,
                                   'current': 5,
                                   'normal': 8}}}]

                          

This means that people's callbacks for the reports will need to change slightly to accommodate the new data format. Since we're still in the pre-1.0 phase I think we can do this, as long as I add a clear warning if the callback fails, so that people can quickly find out how to change their implementation.

If you could try out the version from this branch and let me know if this seems to cover your needs, I'd appreciate it very much.

Thanks.

[francisrsandoval]

Thanks Stan for the quick response to this and the other issues Ive raised. I will be able to test this branch a bit later this week and will provide feedback in several days.

[francisrsandoval]

Hi Stan,

Perhaps I’ve misunderstood something, but this is what I’m trying:

in my test VEN, I add a status report

self.client.add_report(callback=self.collect_status,
                        data_collection_mode='full',
                        resource_id=resource_id,
                        report_name=TELEMETRY_STATUS,
                        sampling_rate=STATUS_SAMPLING_RATE)

and the handler includes

intervals = []
interval = {'dtstart': datetime.now(), 'resource_status': {'online': True, 'manual_override': False, 'capacity': {'min': 0, 'max': 10, 'current': 5, 'normal': 8}}}
intervals.append(interval)

but the generated report is

<?xml version="1.0" encoding="utf-8"?>
<oadr:oadrPayload xmlns:oadr="http://openadr.org/oadr-2.0b/2012/07">
<oadr:oadrSignedObject xmlns:oadr="http://openadr.org/oadr-2.0b/2012/07" xmlns:pyld="http://docs.oasis-open.org/ns/energyinterop/201110/payloads" xmlns:emix="http://docs.oasis-open.org/ns/emix/2011/06" oadr:Id="oadrSignedObject"><oadr:oadrUpdateReport ei:schemaVersion="2.0b" xmlns:ei="http://docs.oasis-open.org/ns/energyinterop/201110"><pyld:requestID>74a4dba6-df37-4bdb-82dd-6c85c6a2c39d</pyld:requestID><oadr:oadrReport xmlns:xcal="urn:ietf:params:xml:ns:icalendar-2.0"><xcal:dtstart><xcal:date-time>dtstart</xcal:date-time></xcal:dtstart><strm:intervals xmlns:strm="urn:ietf:params:xml:ns:icalendar-2.0:stream"><ei:interval><xcal:dtstart><xcal:date-time>dtstart</xcal:date-time></xcal:dtstart><oadr:oadrReportPayload><ei:rID>f78b65d2-8ee4-4724-b529-4bb3d5cddb36</ei:rID><ei:payloadFloat><ei:value>resource_status</ei:value></ei:payloadFloat></oadr:oadrReportPayload></ei:interval></strm:intervals><ei:eiReportID></ei:eiReportID><ei:reportRequestID>9fe03a28-2544-4aec-9acd-d3bd38fc681e</ei:reportRequestID><ei:reportSpecifierID>db73352f-7877-40d9-af76-a6a0a7d2ab76</ei:reportSpecifierID><ei:reportName>TELEMETRY_STATUS</ei:reportName><ei:createdDateTime>2022-04-20T14:57:30.008973Z</ei:createdDateTime></oadr:oadrReport><ei:venID>NHEC_test_01</ei:venID></oadr:oadrUpdateReport></oadr:oadrSignedObject>
</oadr:oadrPayload>

I would expect the interval to contain dtstart and something other than oadrReportPayload to contain resource_status

The VTN replies with

</oadr:oadrPayload>: 400 XML failed validation: Element '{urn:ietf:params:xml:ns:icalendar-2.0}date-time': 'dtstart' is not a valid value of the atomic type '{urn:ietf:params:xml:ns:icalendar-2.0}DateTimeType'. (, line 0)

Seems odd that it chokes on dtstart and not the payload of the interval.

I’m On branch add-status-reports

I’ll keep digging but perhaps you can enlighten me and save me some time :-)

thanks

[francisrsandoval]

I’ve made progress by not specifying data_collection_mode=‘full’ but the ‘capacity’ dict is not being carried through to the XML msg. Am continuing to debug

Frank Sandoval Principal - Pajarito Technologies LLC email: @.*** mobile: 720 338-1988

On Apr 20, 2022, at 9:18 AM, Frank Sandoval @.***> wrote:

Hi Stan,

Perhaps I’ve misunderstood something, but this is what I’m trying:

in my test VEN, I add a status report self.client.add_report(callback=self.collect_status, data_collection_mode='full', resource_id=resource_id, report_name=TELEMETRY_STATUS, sampling_rate=STATUS_SAMPLING_RATE) and the handler includes intervals = [] interval = {'dtstart': datetime.now(), 'resource_status': {'online': True, 'manual_override': False, 'capacity': {'min': 0, 'max': 10, 'current': 5, 'normal': 8}}} intervals.append(interval) but the generated report is

<oadr:oadrPayload xmlns:oadr="http://openadr.org/oadr-2.0b/2012/07 http://openadr.org/oadr-2.0b/2012/07"> <oadr:oadrSignedObject xmlns:oadr="http://openadr.org/oadr-2.0b/2012/07 http://openadr.org/oadr-2.0b/2012/07" xmlns:pyld="http://docs.oasis-open.org/ns/energyinterop/201110/payloads http://docs.oasis-open.org/ns/energyinterop/201110/payloads" xmlns:emix="http://docs.oasis-open.org/ns/emix/2011/06 http://docs.oasis-open.org/ns/emix/2011/06" oadr:Id="oadrSignedObject"><oadr:oadrUpdateReport ei:schemaVersion="2.0b" xmlns:ei="http://docs.oasis-open.org/ns/energyinterop/201110 http://docs.oasis-open.org/ns/energyinterop/201110">pyld:requestID74a4dba6-df37-4bdb-82dd-6c85c6a2c39d</pyld:requestID><oadr:oadrReport xmlns:xcal="urn:ietf:params:xml:ns:icalendar-2.0">xcal:dtstartxcal:date-timedtstart</xcal:date-time></xcal:dtstart><strm:intervals xmlns:strm="urn:ietf:params:xml:ns:icalendar-2.0:stream">ei:intervalxcal:dtstartxcal:date-timedtstart</xcal:date-time></xcal:dtstart>oadr:oadrReportPayloadei:rIDf78b65d2-8ee4-4724-b529-4bb3d5cddb36</ei:rID>ei:payloadFloatei:valueresource_status</ei:value></ei:payloadFloat></oadr:oadrReportPayload></ei:interval></strm:intervals>ei:eiReportID</ei:eiReportID>ei:reportRequestID9fe03a28-2544-4aec-9acd-d3bd38fc681e</ei:reportRequestID>ei:reportSpecifierIDdb73352f-7877-40d9-af76-a6a0a7d2ab76</ei:reportSpecifierID>ei:reportNameTELEMETRY_STATUS</ei:reportName>ei:createdDateTime2022-04-20T14:57:30.008973Z</ei:createdDateTime></oadr:oadrReport>ei:venIDNHEC_test_01</ei:venID></oadr:oadrUpdateReport></oadr:oadrSignedObject> </oadr:oadrPayload>

I would expect the interval to contain dtstart and something other than oadrReportPayload to contain resource_status

The VTN replies with

</oadr:oadrPayload>: 400 XML failed validation: Element '{urn:ietf:params:xml:ns:icalendar-2.0}date-time': 'dtstart' is not a valid value of the atomic type '{urn:ietf:params:xml:ns:icalendar-2.0}DateTimeType'. (, line 0)

Seems odd that it chokes on dtstart and not the payload of the interval.

I’m On branch add-status-reports

I’ll keep digging but perhaps you can enlighten me and save me some time :-)

thanks

Frank Sandoval Principal - Pajarito Technologies LLC email: @.*** @.***> mobile: 720 338-1988

On Apr 9, 2022, at 4:51 PM, Stan Janssen @.*** @.***>> wrote:

I implemented a first version of the TELEMETRY_STATUS reporting functionality in a separate branch https://github.com/OpenLEADR/openleadr-python/tree/add-status-reports. The interface is as described above.

It does come with a small change to how the VTN hands off data to the report callback function: it used to supply a list of (datetime.datetime, value) items that corresponded with the measured intervals; it now supplies a list of dicts that contain the report data in the following format:

{'dtstart': datetime.datetime(...), 'value': 1.00} Or, for the status reports:

{'dtstart': datetime.datetime(...), 'resource_status': {'online': True, 'manual_override': False, 'capacity': {'min': 0, 'max': 10, 'current': 5, 'normal': 8}}}

This means that people's callbacks for the reports will need to change slightly to accommodate the new data format. Since we're still in the pre-1.0 phase I think we can do this, as long as I add a clear warning if the callback fails, so that people can quickly find out how to change their implementation.

If you could try out the version from this branch and let me know if this seems to cover your needs, I'd appreciate it very much.

Thanks.

— Reply to this email directly, view it on GitHub https://github.com/OpenLEADR/openleadr-python/issues/108#issuecomment-1094137161, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQJNPUJS6CDDEQ2X5ZGT543VEICW3ANCNFSM5SXIGAHQ. You are receiving this because you authored the thread.

[stan-janssen]

The VEN handler must look like this:

def status_callback_full():
    return True, False, {'capacity': {'min': 0,
                                      'max': 10,
                                      'current': 5,
                                      'normal': 8}}

And the VTN handler must accept a list of dicts:

[{'dtstart': datetime.datetime(...),
  'resource_status': {'online': True,
                      'manual_override': False,
                      'capacity': {'min': 0,
                                   'max': 10,
                                   'current': 5,
                                   'normal': 8}}},
 {'dtstart': datetime.datetime(...),
  'resource_status': {'online': True,
                      'manual_override': False,
                      'capacity': {'min': 0,
                                   'max': 10,
                                   'current': 5,
                                   'normal': 8}}}]

You should, indeed, not use data_collection_mode = full for now.

[francisrsandoval]

Thank you Stan, I’ve got things working now!

Frank Sandoval Principal - Pajarito Technologies LLC email: @.*** mobile: 720 338-1988

On Apr 20, 2022, at 12:59 PM, Stan Janssen @.***> wrote:

{'capacity': {'min': 0, 'max': 10, 'current': 5, 'normal': 8}}

[stan-janssen]

All right, I will try to polish this up a bit, make sure the documentation is clear, and then we can move this into the next version.

(I actually believe that there is a requirement that all resources have this status report, so maybe there should be a more abstract notion of registering "resources" in an OpenLEADR VEN, instead of registering the reports and possible event handlers separately, but that's for another time.)

[francisrsandoval]

Thanks Stan, yes, it appears status reports are good

Frank Sandoval Principal - Pajarito Technologies LLC email: @.*** mobile: 720 338-1988

On May 2, 2022, at 1:58 PM, Stan Janssen @.***> wrote:

All right, I will try to polish this up a bit, make sure the documentation is clear, and then we can move this into the next version.

(I actually believe that there is a requirement that all resources have this status report, so maybe there should be a more abstract notion of registering "resources" in an OpenLEADR VEN, instead of registering the reports and possible event handlers separately, but that's for another time.)

— Reply to this email directly, view it on GitHub https://github.com/OpenLEADR/openleadr-python/issues/108#issuecomment-1115302810, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQJNPUJMOMIOCXI2NQD3S7LVIAXXJANCNFSM5SXIGAHQ. You are receiving this because you authored the thread.

[juantxorena]

This works also for me. Is it going to be merged in the main branch?

[perdue]

Discovering this library now. It sounds like you addressed this issue and implemented a fix. @stan-janssen, do you plan on merging this feature branch into main?