Bad gap-filling on SWIS rooftop

[nc9]

[simonhac]

lots of data is missing and we appear to be gapfilling.

[simonhac]

i think we shouldn't gapfill any solar data across 30 minute boundaries on the FE.

[nc9]

@chienleng I think the FE should should none here when it's empty

I just re-ran the old APVI endpoint and it has those values again.

[nc9]

[nc9]

https://twitter.com/OpenNem/status/1592334328780161024

[simonhac]

@nc9 so APVI fixed the bad data?

a few issues i think:

• APVI sending bad data — sh*t happens
• APVI sending null (missing data) where they mean zero (0 MW) — should we talk to them about this?
• our FE gap-filling over null — do we want this?

[nc9]

I'll leave this open even with that patch above. The data seems to shift from underneath us - I'll collect a bit more data around this but we have more sanity checking now

APVI returns no interval timestamp for periods that are assumed to be 0 - the SQL out is casting those to 0 now

[nc9]

I'm going to open another issue with APVI data issues - I've got a monitoring script now and some sanity-checks

[andrewbusch7]

Quite a major one from 1 Dec 4 PM through to 3 Dec 4 AM. That's some pretty bold gap-filling, I too would prefer seeing zeroes in such cases. Could do the linear gap fill only if values are on the same day for solar, and otherwise zero.

[nc9]

That's some pretty bold gap-filling,

I believe that is less gap-filling and more graphing the line down to the next data point

I too would prefer seeing zeroes in such cases

100% agree. I have a changeset for this weeks release for this case

[nc9]

https://opennem.org.au/energy/wem/?range=7d&interval=30m

We eventually got complete data from APVI but this is more about how we handle cases where it fails

[simonhac]

@nc9 what did you actually do to fix this? the underlying cause, if i recall correctly was twofold:

• FE interpolates when it shouldn't. did we fix this?
• data doesn't differentiate between 0 MW and missing data.