Use ETag and If-Not-Modified

[lkiesow]

To reduce load for the Scheduler…

[lkiesow]

Note to myself: http://stackoverflow.com/a/472243/2352895

[lkiesow]

We need to be careful with Opencast's handling of ETags together with the use of the cutoff parameter. Cutoff may limit data received and following requests will return a NOT MODIFIED while not all data have been transmitted.

[smarquard]

So as I understand it, Opencast should only return NOT MODIFIED only if the calendar has not changed in any way since the last request. So there shouldn't be an issue combining cutoff with ETags?

Seems like the worst-case scenario is a false-positive, i.e. if you request with a cutoff date of 7 days ahead, and the calendar has modified because a recording was added 10 days ahead, you could still get a whole new calendar for the next 7 days even though it's not necessary.

But that's still better than getting a new calendar on every request.

[smarquard]

In Opencast 4.x getting a calendar is a relatively expensive request, even an empty calendar as in the example below which takes 2.2s db query time to produce zero rows.

So for pyca with OC 4.x, using ETag / not-modified is an important optimization for CAs.

# Time: 2018-01-15T14:24:20.357129Z
# User@Host: opencast[opencast] @ srvslsopc004.uct.ac.za [137.158.158.211]  Id: 502817
# Query_time: 2.271741  Lock_time: 0.000996 Rows_sent: 0  Rows_examined: 151745
SET timestamp=1516026260;
SELECT t0.id, t0.archival_date, t0.availability, t0.mediapackage_id, t0.mediapackage_xml, t0.organization_id, t0.owner, t0.series_id, t0.version, t1.id, t1.val_bool, t1.val_date, t1.val_long, t1.mediapackage_id, t1.namespace, t1.property_name, t1.val_string FROM mh_assets_snapshot t0 LEFT OUTER JOIN mh_assets_properties t1 ON (((((t1.mediapackage_id = t0.mediapackage_id) AND ((t1.namespace = 'org.opencastproject.scheduler') AND (t1.property_name = 'agent'))) OR ((t1.mediapackage_id = t0.mediapackage_id) AND ((t1.namespace = 'org.opencastproject.scheduler') AND (t1.property_name = 'start')))) OR ((t1.mediapackage_id = t0.mediapackage_id) AND ((t1.namespace = 'org.opencastproject.scheduler') AND (t1.property_name = 'end')))) OR ((t1.mediapackage_id = t0.mediapackage_id) AND (t1.namespace = 'org.opencastproject.scheduler.ca.configuration'))) WHERE (((((((t0.organization_id = 'mh_default_org') AND EXISTS (SELECT 1 FROM mh_assets_properties t2 WHERE ((t0.mediapackage_id = t2.mediapackage_id) AND (t2.namespace = 'org.opencastproject.scheduler'))) ) AND EXISTS (SELECT 1 FROM mh_assets_properties t3 WHERE ((((t0.mediapackage_id = t3.mediapackage_id) AND (t3.namespace = 'org.opencastproject.scheduler')) AND (t3.property_name = 'optout')) AND (t3.val_bool = 0))) ) AND ((NOT EXISTS (SELECT 1 FROM mh_assets_properties t4 WHERE (((t0.mediapackage_id = t4.mediapackage_id) AND (t4.namespace = 'org.opencastproject.scheduler.trx')) AND (t4.property_name = 'transaction_id')))  AND (t0.version = (SELECT MAX(t5.version) FROM mh_assets_snapshot t5 WHERE (t5.mediapackage_id = t0.mediapackage_id)))) OR (EXISTS (SELECT 1 FROM mh_assets_properties t6 WHERE (((t0.mediapackage_id = t6.mediapackage_id) AND (t6.namespace = 'org.opencastproject.scheduler.trx')) AND (t6.property_name = 'transaction_id')))  AND EXISTS (SELECT 1 FROM mh_assets_properties t7 WHERE ((((t0.mediapackage_id = t7.mediapackage_id) AND (t7.namespace = 'org.opencastproject.scheduler')) AND (t7.property_name = 'version')) AND (t0.version = t7.val_long))) ))) AND EXISTS (SELECT 1 FROM mh_assets_properties t8 WHERE ((((t0.mediapackage_id = t8.mediapackage_id) AND (t8.namespace = 'org.opencastproject.scheduler')) AND (t8.property_name = 'end')) AND (t8.val_date >= '2018-01-15 15:24:18.084'))) ) AND EXISTS (SELECT 1 FROM mh_assets_properties t9 WHERE ((((t0.mediapackage_id = t9.mediapackage_id) AND (t9.namespace = 'org.opencastproject.scheduler')) AND (t9.property_name = 'agent')) AND (t9.val_string = 'jordan1c'))) ) AND EXISTS (SELECT 1 FROM mh_assets_properties t10 WHERE ((((t0.mediapackage_id = t10.mediapackage_id) AND (t10.namespace = 'org.opencastproject.scheduler')) AND (t10.property_name = 'start')) AND (t10.val_date <= '2018-01-29 16:24:18'))) );