ED tests are broken

[robkooper]

[ashiklom]

@robkooper Can you provide some more info on what we meant by this? Was it that our tests for checking that ED runs successfully finished don't work -- i.e. the workflow is reporting that the runs finished, when in fact they failed somewhere in the ED execution?

[robkooper]

Running the following on the VM from the pecan/tests folder

../web/workflow.R pecan32.ed.xml

Results in the following error:

2017-08-22 13:26:34 INFO   [PEcAn.settings::read.settings] :
   Loading inpufile= pecan32.ed.xml
2017-08-22 13:26:34 INFO   [fix.deprecated.settings] :
   Fixing deprecated settings...
2017-08-22 13:26:34 INFO   [fix.deprecated.settings] :
   settings$run$dbfiles is deprecated. uwe settings$database$dbfiles
   instead
2017-08-22 13:26:34 INFO   [fix.deprecated.settings] :
   settings$run$host is deprecated. uwe settings$host instead
2017-08-22 13:26:34 INFO   [update.settings] :
   Fixing deprecated settings...
2017-08-22 13:26:34 INFO   [check.settings] : Checking settings...
Loading required package: RPostgreSQL
2017-08-22 13:26:34 INFO   [check.database] :
   Successfully connected to database : PostgreSQL bety bety localhost bety
   FALSE
2017-08-22 13:26:34 WARN   [check.database.settings] :
   Will not write runs/configurations to database.
2017-08-22 13:26:35 WARN   [check.bety.version] :
   Last migration 20170415183619 is more recent than expected
   20141009160121. This could result in PEcAn not working as expected.
2017-08-22 13:26:35 INFO   [fn] :
   Setting ensemble size to 1.
2017-08-22 13:26:35 INFO   [fn] :
   No start date passed to ensemble - using the run date ( 2006 ).
2017-08-22 13:26:35 INFO   [fn] :
   No end date passed to ensemble - using the run date ( 2006 ).
2017-08-22 13:26:35 INFO   [fn] :
   Setting site name to EBI Energy farm
2017-08-22 13:26:35 INFO   [fn] :
   Setting site lat to 40.0637
2017-08-22 13:26:35 INFO   [fn] :
   Setting site lon to -88.202
2017-08-22 13:26:35 INFO   [check.model.settings] :
   Setting model id to 1000000003
2017-08-22 13:26:35 WARN   [check.model.settings] :
   Model revision specified in settings file does not match database.
2017-08-22 13:26:35 INFO   [check.model.settings] :
   Option to delete raw model output not set or not logical. Will keep all
   model output.
2017-08-22 13:26:35 WARN   [PEcAn.DB::dbfile.file] :
   no files found for 1000000003 in database
2017-08-22 13:26:35 WARN   [check.settings] :
   settings$database$dbfiles pathname pecan/dbfiles is invalid

   placing it in the home directory /home/kooper
2017-08-22 13:26:35 INFO   [fn] :
   path /home/kooper/sites/ebifarm/ebifarm.lat40.0lon-88.0.css
2017-08-22 13:26:36 INFO   [fn] :
   path /home/kooper/sites/ebifarm/ebifarm.lat40.0lon-88.0.css
2017-08-22 13:26:36 WARN   [PEcAn.DB::dbfile.id] :
   no id found for /home/kooper/ed_inputs/glu in database
2017-08-22 13:26:36 INFO   [fn] :
   path /home/kooper/ed_inputs/glu
2017-08-22 13:26:36 INFO   [fn] :
   path /home/kooper/ed_inputs/glu
2017-08-22 13:26:36 INFO   [fn] :
   path /home/kooper/sites/ebifarm/ED_MET_DRIVER_HEADER
2017-08-22 13:26:36 INFO   [fn] :
   path /home/kooper/sites/ebifarm/ED_MET_DRIVER_HEADER
2017-08-22 13:26:36 INFO   [fn] :
   path /home/kooper/sites/ebifarm/ebifarm.lat40.0lon-88.0.pss
2017-08-22 13:26:36 INFO   [fn] :
   path /home/kooper/sites/ebifarm/ebifarm.lat40.0lon-88.0.pss
2017-08-22 13:26:36 INFO   [fn] :
   path /home/kooper/sites/ebifarm/ebifarm.lat40.0lon-88.0.site
2017-08-22 13:26:36 INFO   [fn] :
   path /home/kooper/sites/ebifarm/ebifarm.lat40.0lon-88.0.site
2017-08-22 13:26:36 INFO   [fn] :
   path /home/kooper/faoOLD/FAO_
2017-08-22 13:26:36 INFO   [fn] :
   path /home/kooper/faoOLD/FAO_
2017-08-22 13:26:36 WARN   [PEcAn.DB::dbfile.id] :
   no id found for /home/kooper/ed_inputs in database
2017-08-22 13:26:36 INFO   [fn] :
   path /home/kooper/ed_inputs
2017-08-22 13:26:36 INFO   [fn] :
   path /home/kooper/ed_inputs
2017-08-22 13:26:36 INFO   [fn] :
   path /home/kooper/oge2OLD/OGE2_
2017-08-22 13:26:36 INFO   [fn] :
   path /home/kooper/oge2OLD/OGE2_
2017-08-22 13:26:36 INFO   [check.workflow.settings] :
   output folder = /home/kooper/pecan/tests/pecan
2017-08-22 13:26:36 INFO   [check.settings] :
   Storing pft ebifarm.c4grass.doe_vd in
   /home/kooper/pecan/tests/pecan/pft/ebifarm.c4grass.doe_vd
2017-08-22 13:26:36 WARN   [PEcAn.settings::write.settings] :
   File already exists [ /home/kooper/pecan/tests/pecan/pecan.CHECKED.xml ]
   file will be overwritten
[1] "/home/kooper/pecan/tests/pecan/pecan.CHECKED.xml"
[1] TRUE
2017-08-22 13:26:36 DEBUG  [PEcAn.utils::logger.debug] :
   do.conversion outdir /home/kooper/pecan/dbfiles
2017-08-22 13:26:36 INFO   [PEcAn.utils::logger.info] : PROCESSING:  met
2017-08-22 13:26:36 INFO   [PEcAn.utils::logger.info] :
   calling met.process: /home/kooper/sites/ebifarm/ED_MET_DRIVER_HEADER
2017-08-22 13:26:36 WARN   [PEcAn.data.atmosphere::met.process] :
   No met source specified
2017-08-22 13:26:36 WARN   [PEcAn.data.atmosphere::met.process] :
   Assuming source CFmet
     scale format.id    format.name      format.mimetype format.inputtype
1 regional        33 CF Meteorology application/x-netcdf               nc
2017-08-22 13:26:36 WARN   [read.register] :
   Region site id is not defined
$download.raw
[1] FALSE

$met2cf
[1] FALSE

$standardize
[1] FALSE

$met2model
[1] FALSE

$id.name
[1] "model.id"

2017-08-22 13:26:36 INFO   [PEcAn.utils::logger.info] :
   ready.id 7 pecan.ncsa.illinois.edu
2017-08-22 13:26:36 INFO   [PEcAn.utils::logger.info] :
   model.file = /home/kooper/sites/ebifarm/ED_MET_DRIVER_HEADER
   /home/kooper/sites/ebifarm/ED_MET_DRIVER_HEADER 7 CFmet
2017-08-22 13:26:36 WARN   [PEcAn.utils::logger.warn] :
   Connection created outside of PEcAn.db package
2017-08-22 13:26:36 DEBUG  [PEcAn.utils::logger.debug] :
   updated met path: /home/kooper/sites/ebifarm/ED_MET_DRIVER_HEADER
2017-08-22 13:26:36 INFO   [PEcAn.utils::logger.info] : PROCESSING:  veg
2017-08-22 13:26:36 INFO   [PEcAn.utils::logger.info] : PROCESSING:  soil
2017-08-22 13:26:36 INFO   [PEcAn.utils::logger.info] : PROCESSING:  pss
2017-08-22 13:26:36 INFO   [PEcAn.utils::logger.info] : PROCESSING:  css
2017-08-22 13:26:36 INFO   [PEcAn.utils::logger.info] : PROCESSING:  site
2017-08-22 13:26:36 INFO   [PEcAn.utils::logger.info] : PROCESSING:  lu
2017-08-22 13:26:36 INFO   [PEcAn.utils::logger.info] : PROCESSING:  thsum
2017-08-22 13:26:37 WARN   [dbfile.check] :
   Multiple Valid Files found on host machine. Returning last updated
   record.
2017-08-22 13:26:37 INFO   [PEcAn.utils::logger.info] :
   /home/kooper/pecan/dbfiles/posterior/99000000003 trait.data.Rdata
2017-08-22 13:26:37 DEBUG  [PEcAn.utils::logger.debug] :
   Checking if trait data has changed
2017-08-22 13:26:37 INFO   [PEcAn.utils::logger.info] :
   /home/kooper/pecan/dbfiles/posterior/99000000003 prior.distns.Rdata
2017-08-22 13:26:37 DEBUG  [PEcAn.utils::logger.debug] :
   Checking if priors have changed
Warning in if (file.exists(files$file_path[[id]], files$file_name[[id]])) { :
  the condition has length > 1 and only the first element will be used
2017-08-22 13:26:37 INFO   [PEcAn.utils::logger.info] :
   /home/kooper/pecan/dbfiles/posterior/99000000003 species.csv
2017-08-22 13:26:37 DEBUG  [PEcAn.utils::logger.debug] :
   Checking if species have changed
2017-08-22 13:26:37 INFO   [PEcAn.utils::logger.info] :
   Reusing existing files from posterior 99000000003 for
   ebifarm.c4grass.doe_vd
2017-08-22 13:26:37 WARN   [PEcAn.settings::write.settings] :
   File already exists [ /home/kooper/pecan/tests/pecan/pecan.TRAIT.xml ]
   file will be overwritten
2017-08-22 13:26:37 INFO   [FUN] :
   -------------------------------------------------------------------
2017-08-22 13:26:37 INFO   [FUN] :
   Running meta.analysis for PFT: ebifarm.c4grass.doe_vd
2017-08-22 13:26:37 INFO   [FUN] :
   -------------------------------------------------------------------
Error in paste(c(msg, ...), collapse = " ") : object 'i' not found
Calls: <Anonymous> ... lapply -> FUN -> logger.warn -> logger.message -> paste

[ashiklom]

So ED in general seems to run fine on the VM, at least with a simple configuration using existing met without conversion. This specific test has two problems:

1. The ebifarm.c4grass.doe_vd PFT is not in the ED PFT mappings, which causes ED write configs to fail. The workaround here is to add it manually to the pftmappings.csv file, but that seems extremely unsatisfying and hacky. I feel like we've run up against this problem before. @mdietze @serbinsh Any thoughts on how to better assign default ED parameters using BETY? Perhaps a machine-readable (JSON? XML? YAML?) in the PFT notes in BETY? This is related to @ankurdesai issue #1261 .

2. The ED executable was wrong. ed2.r82 didn't exist -- only ed2.git is available on the VM.

3. The pss file for this run is ill formed. The columns are in the wrong order, and have the wrong heading names compared to other pss files that work. I propose we just change this test to a css/pss file that works (e.g. Niwot Ridge) and purge the bad css/pss file from the VM and BETY.

[mdietze]

I think that there are two high-level options:

1. just dump the test as a whole, since no one runs it and we're trying to shift to using the Benchmark module instead for integration tests

2. If we actually fix it then:

• add ebifarm.c4grass.doe_vd to pftmappings -- yes it's inelegant, but it's also the current standard practice for this model, and this isn't the issue to reinvent that
• update executable
• fix pss/css rather than switching to a different site (need tests at multiple sites & PFTs)

[ashiklom]

In your second point, I can do the first bullet but the latter 2 are issues with the VM, not with the PEcAn, so @robkooper will have to fix those. I can pass along the fixed ED input files for the site, so all Rob will have to do is copy them onto the VM.

[ashiklom]

Actually, I guess the executable is a change in the XML, and I already changed that to a model ID, which correctly points to the ED git revision executable.

[ashiklom]

Closed by #1619

[Aariq]

ED tests still skipped referencing this issue, reopening.

Currently broken tests:

• test.model2netcdf.ED2.R
• test.write.configs.ed.R

[Aariq]

Can't re-enable tests until #2981 is fixed