Veracity of data

[rmward61]

Checking veracity of data

Need to incorporate checks on the monitored data coming in from the platform for two reasons:

• If the data aren't available the models cannot run
• If the data are incorrect the models will give unrealistic predictions

Data availability

Straightforward to check, but the different models require different datasets and quantities of data so care needs to be taken that all requirements are checked. Here is a summary table showing the data required for each model:

Model	Data required	Source	Impact	Solution
Static
Dynamic (GES)	External T, RH for 10 days prior to calibration	Zensie weather	No data - cannot run model	Error message
			Partial data - model might be OK with less than 10 days, to be checked	Run model with fewer days but generate warning message
	Monitored RH at centre of farm at 12 hourly intervals over 10 days	Zensie sensor 16_B2 (check)	No data - cannot run calibration	Error message
			Partial data - will run OK provided not too much missing and first point not missing	Check first calibration point exists, check sufficient data points for calibration (how many?), could use data from different sensor. Warning message

Incorrect data

'Incorrect' can mean several things:

• Values completely outside reasonable limits ( -ve or zero values for example, or RH > 100%)
• Values outside reasonable limits for the season e.g. high temperatures in January
• Value distribution unrealistic i.e. sensors physically located close together giving different readings
• Timestamp errors (e.g. incorrect identification of BST/GMT)
• Errors in model parameters (changes to number of dehumidifiers, lighting power etc)

and there can be many different causes, for example:

• Sensors moved
• Sensors switched off
• Calibration required
• Failure of connection to database
• Lack of communication

The following table summarises potential issues and remedies.

Model	Data	Issue	Source	Impact	Solution
Static
Dynamic (GES)	External Temperature	Value < lower limit (-15?) or > upper limit (40)	Zensie	Values unbelievable	Data not useable - error message
	External RH	Value < lower limit (0%) or > upper limit (100%)	Zensie	Values unbelievable	Data not useable - error message
	External Temperature	Value outside seasonal limits	Zensie	Values unbelievable	Warning - investigate
	External RH	Value outside seasonal limits	Zensie	Values unbelievable	Warning - investigate
	Monitored RH	Value < lower limit (0%) or > upper limit (100%)	Zensie	Values unbelievable	Data not useable - error message
	Monitored RH	Value < upper lower limit (30% ?) or > lower upper limit (90% ? )	Zensie	Values unlikely	Check against surrounding monitors - if consistent raise warning, if inconsistent do not use for calibration - error message or switch to nearby monitor (need to think this through)
	Monitored RH	Difference between monitors incompatible (is this something we can identify?)	Zensie	Suggests error on one or other monitor	Warning - investigate?
	Model parameters	Difference between model and reality	Farm	Model incorrect	Edit model - but how to identify?