TG2-AMENDMENT_DATEIDENTIFIED_STANDARDIZED

[iDigBioBot]

TestField	Value
GUID	39bb2280-1215-447b-9221-fd13bc990641
Label	AMENDMENT_DATEIDENTIFIED_STANDARDIZED
Description	Proposes an amendment to the value of dwc:dateIdentified to a valid ISO date.
TestType	Amendment
Darwin Core Class	dwc:Identification
Information Elements ActedUpon	dwc:dateIdentified
Information Elements Consulted
Expected Response	INTERNAL_PREREQUISITES_NOT_MET if dwc:dateIdentified is bdq:Empty; AMENDED if the value of dwc:dateIdentified is not a properly formatted ISO 8601 date but is unambiguous and altered to be a valid ISO 8601 date; otherwise NOT_AMENDED.
Data Quality Dimension	Conformance
Term-Actions	DATEIDENTIFIED_STANDARDIZED
Parameter(s)
Source Authority
Specification Last Updated	2024-09-16
Examples	[dwc:dateIdentified="2021-28-10": Response.status=AMENDED, Response.result=dwc:dateIdentified="2021-10-28", Response.comment="dwc:dateIdentified assuming dwc:year, dwc:day and dwc:month"]
	[dwc:dateIdentified="21-10-28": Response.status=NOT_AMENDED, Response.result=, Response.comment="dwc:dateIdentified contains ambiguous values. It could be dd-mm-yy or yy-mm-dd"]
Source	Kurator
References	ISO (2019) ISO 8601-1:2019(en) Date and time — Representations for information interchange — Part 1: Basic rules. https://www.iso.org/obp/ui/ Wikipedia (2020) ISO 8601. https://en.wikipedia.org/wiki/ISO_8601 Library of Congress (2019) Extended Date/Time Format (EDTF). https://www.loc.gov/standards/datetime/
Example Implementations (Mechanisms)	Kurator:event_date_qc
Link to Specification Source Code	https://github.com/FilteredPush/event_date_qc/blob/1abbd3f02eb6c28129764defab78f72156972864/src/main/java/org/filteredpush/qc/date/DwCEventDQ.java#L489
Notes	We reference Wikipedia for the ISO standard because the standard documents are not free.

[iDigBioBot]

Comment by Paul Morris (@chicoreus) migrated from spreadsheet: Corresponding ammendment for IDENTIFIED_DATE_INVALID (VALIDATION_DATEIDENTIFIED_INVALID)

[chicoreus]

There appears to be a copy/paste error in the specification,

EXTERNAL_PREREQUESITES_NOT_MET if the specified target source authority was not found;

This test does not involve a query on any external source authority such as a list of taxonomic names.

[ArthurChapman]

@chicoreus This probably refers to the ISO 8601-1:2019 that is needed. But you probably don't need to go to the actual Standard to use the format, so probably unnecessary. This may apply to other date tests?

[Tasilee]

If it doesn't have a Parameter, then there is no "specified target authority"?

[chicoreus]

EXTERNAL{PREREQUISITES_NOT_MET should signal that the test must connect to an external resource in order to perform the test, this test would be expected to be implemented strictly with local code that doesn't need to do a network lookup each time the test is run, so no. need for this assertion in the specification.

@Tasilee A test might have a parameter and not need to connect to an external source (such as earliest date as a parameter), or a test might specify some authority to be used by implementors (such as the ISO date format specified here), but not need to connect to an external source on each run, and it might do so with or without taking the source authority as a parameter (though I expect our likely taxonomic cases that would take a taxonmic authority as a source would typically need to do a lookup for each test run - the relationship is logical, but isn't necessaraly present).

[ArthurChapman]

@chicoreus your comment would apply to all that are tagged ISO/DCMI Standard. To date we have only used the Parameter field to define default minimum and maximums and source Authorities (the later using bdq:sourceAuthority (defalt=xxxxx)) in Paramaterized tests. Perhaps we should extend that to include standard source authorities such as ISO and DCMI Standards using bdq:sourceAuthority=ISO 8601-1:2019 etc. under the Parameter field ?? What do others think?

[Tasilee]

I agree with @chicoreus: If there is no (EXTERNAL) source authority, then we remove that part of the Expected Responses. Other comments?

@ArthurChapman: Hopefully quoting @chicoreus - If there is only one obvious bdq:sourceAuthority or maybe value, it is not a Parameter?

Then it is a demarcation between a bdq:sourceAuthority Parameter API-type lookup vs a Reference to a standard that the test relies on (as the latter is here). The ISO/DCMI STANDARD flag is visible evidence of the use of a particular standard but is not needed as a Parameter.

[chicoreus]

@ArthurChapman Parameter should not be specified for cases like this where there is a standardized source authority that implementors would be expected to embed in their code. Parameter should only be used when different use cases would call for different limits (such as a national data set wishing to use elevation/depth limits that apply to their country and a national taxonomic authority file or species list).

Issue looking good, removing the needs work label.

[chicoreus]

Actually, making one change (adding unambigous) to conform with language in #61

From:

INTERNAL_PREREQUISITES_NOT_MET if the field dwc:dateIdentified was not present or is EMPTY; AMENDED if the value of dwc:dateIdentified was altered to conform with the ISO 8601-1:2019 date format; otherwise NOT_CHANGED

To:

INTERNAL_PREREQUISITES_NOT_MET if the field dwc:dateIdentified was not present or is EMPTY; AMENDED if the value of dwc:dateIdentified was altered to unambiguously conform with the ISO 8601-1:2019 date format; otherwise NOT_CHANGED

[Tasilee]

Due to recent discussions, changed

INTERNAL_PREREQUISITES_NOT_MET if dwc:dateIdentified is EMPTY; AMENDED the value of dwc:dateIdentified if it can be unambiguously interpreted as valid using bdq:sourceAuthority; otherwise NOT_AMENDED

to

INTERNAL_PREREQUISITES_NOT_MET if dwc:dateIdentified is EMPTY; AMENDED the value of dwc:dateIdentified if can be unambiguously interpreted as valid ISO 8601-1 date; otherwise NOT_AMENDED

...and also removed bdq:sourceAuthority

[tucotuco]

This seems wrong to me now. "dwc:dateIdentified if can be unambiguously interpreted as valid ISO 8601-1 date" says to me that dwc:dateIdentified already is a valid ISO 8601-1 date, in which case no amendment would be needed. How about,

"INTERNAL_PREREQUISITES_NOT_MET if dwc:dateIdentified is EMPTY; AMENDED the value of dwc:dateIdentified if it can be unambiguously formatted as a valid ISO 8601-1 date; otherwise NOT_AMENDED"

[ArthurChapman]

Interesting suggestion @tucotuco - I think formatted is better than interpreted here.

One decision we have to make - discussed briefly elsewhere is do we use: ISO 8601 ISO 8601-1 ISO 8601-1:2019 ISO 8601-1:2019/Amd 1:2022

For the tests, I am inclined to just use either ISO 8601 or ISO 8601-1 this saves having to change all the tests if there is a later version of ISO 8601, and we assume that everyone will use the latest version. Incidentally the Amendment allows for midnight to be either 00:00 or 24:00 (previous versions allowed both, but ISO 8601:2019 only allowed 00:00). ISO 8601-1 is the Rules ISO 8601-2 is the Extensions

[tucotuco]

I like "the latest version of ISO 8601-1".

[ArthurChapman]

I like that solution @tucotuco

[Tasilee]

@chicoreus - do you agree with that re-wording to "the latest version of ISO 8601-1" to be applied across all relevant tests?

[Tasilee]

Do we have an issue as the reference remains specific:

"ISO (2019). ISO 8601-1:2019(en) Date and time — Representations for information interchange — Part 1: Basic rules (https://www.iso.org/obp/ui/)

Would it be better to simplify to using "ISO 8601-1" in the Expected Responses and refer to "latest version" in the References?

"ISO 8601-1: Latest version (currently 2019(en)) Date and time — Representations for information interchange — Part 1: Basic rules (https://www.iso.org/obp/ui/)"

?

[ArthurChapman]

I like "the latest version of ISO 8601-1" that allows the use of amendments etc. but I'd accept either way

[tucotuco]

I agree with Arthur. Sufficient that the user knows where to go to see what we're talking about while minimizing standard maintenance. I think this would be a good thing to recommend in DwC term change requests as well.

[Tasilee]

I have amended the Reference and corrected a typo in the Expected response. Please check.

[Tasilee]

I've edited the Expected Response according to @tucotuco suggestion:

From

INTERNAL_PREREQUISITES_NOT_MET if dwc:dateIdentified is EMPTY; AMENDED the value of dwc:dateIdentified if it can be unambiguously formatted as a valid ISO 8601-1 date; otherwise NOT_AMENDED

to

INTERNAL_PREREQUISITES_NOT_MET if dwc:dateIdentified is EMPTY; AMENDED the value of dwc:dateIdentified if it was unambiguously formatted as a valid ISO 8601-1 date; otherwise NOT_AMENDED

[ArthurChapman]

Sorry - I am not sure that the new wording reads well. How about:

INTERNAL_PREREQUISITES_NOT_MET if dwc:dateIdentified is EMPTY; AMENDED the value of dwc:dateIdentified if it could be unambiguously formatted as a valid ISO 8601-1 date; otherwise NOT_AMENDED

[ArthurChapman]

After discussion with @Tasilee - I am not sure, but can accept was. It just doesn't seem to read well. I am not sure with "was", if unambiguously is in the right place - my thinking is that if it is unambiguous (i.e. 3 April rather than 03-04) then you format it there is then nothing ambiguous or unambiguous about the formatting - if it is ambiguous (i.e. 03-04) then it is NOT_AMENDED

INTERNAL_PREREQUISITES_NOT_MET if dwc:dateIdentified is EMPTY; AMENDED the value of dwc:dateIdentified if it was unambiguous by formatting as a valid ISO 8601-1 date; otherwise NOT_AMENDED

[tucotuco]

Counter offer...

INTERNAL_PREREQUISITES_NOT_MET if dwc:dateIdentified is EMPTY; AMENDED the value of dwc:dateIdentified if it was unambiguous and formatted as a valid ISO 8601-1 date; otherwise NOT_AMENDED

[Tasilee]

I have updated the Expected Response (as suggested) and the ISO reference (and while it looks odd, it works).

[ArthurChapman]

What is the thumbs down symbol in the middle of the reference?

[Tasilee]

Some form of markdown I presume, but as I said, it works as a link.

[ArthurChapman]

It appears that ":-" appears as a thumbs down sign in GitHub

[chicoreus]

Current phrasing is not readily iterpretable by implementors. It implies AMENDED if the value already conforms, not if it has been changed to conform. Amendments should be explicit about changing values. The text "if it was unambiguous and formatted" means to an implementor that the text was, past tense, prior to any action, unambigous and formatted (noun) in the expected form. We should be using explicit "altered to conform" not "formatted, which could be read as either the data having had that that state in the past or having been changed to that state. Other amendments need review to make sure that this form of ambiguity has not crept into them.

Very strongly recommend that we return to the previous phrasing.

Changing back from:

INTERNAL_PREREQUISITES_NOT_MET if dwc:dateIdentified is EMPTY; AMENDED the value of dwc:dateIdentified if it was unambiguous and formatted as a valid ISO 8601-1 date; otherwise NOT_AMENDED

to:

INTERNAL_PREREQUISITES_NOT_MET if dwc:dateIdentified is EMPTY; AMENDED if the value of dwc:dateIdentified was altered to unambiguously conform with the ISO 8601-1:2019 date format; otherwise NOT_AMENDED

[chicoreus]

@Tasilee my feeling is that we should reference ISO 8601-1:2019, not the latest version, as the latest version could change the nature of the specifications, with the potential of making test cases and implementations diverge. Referencing a standard set of data values (as in a list taxon names) carries with it the expectation that the data will change, and thus that test cases that test implementations are validated against may need to be changed over time without either the specification for a test or implementations of the test changing, but referencing a standard for the format of data carries the expectation that the format won't change, and that neither test cases nor implementation should change over time without a change in the specification for the test.

[chicoreus]

Whether that is a practical issue for latest version of ISO 8601-1 vs ISO 8601-1:2019 is another question. @ArthurChapman 's note about introduction of validity of 24:00 in a subsequent amendment is relevant....