normalize timestamps and datestamps

[burritojustice]

in the CLI, select columns/properties with date strings, and convert to unix millisecond timestamps and ISO 8601 strings using a --date option.

here xyz upload -f file.[csv|geojson] --date start_date,end_date

• convert to Unix millisecond timestamps with Date.parse() and add this as a new property called 'xyz_timestamp_propertyname`

• convert to ISO 8601 using toISOString -- xyz_iso8601_propertyname

• break out time units as tags (day of week, month, etc, more details below)

[burritojustice]

also parse a property date into tags

provide an option to parse data elements into useful tags, both individually, and in combination. (use ISO 8601 conventions whenever possible)

presuming we are looking at a property called "start_date":

here xyz upload -f file.[csv/geojson] --datetag start_date

• year: start_date_year@2020
• month: start_date_month@december
• year-month: start_date_year_month@2020-02
• week of year: start_date_weekofyear@W02
• year-week start_date_year_weekofyear@2020-W02
• day of week: start_date_weekday@tuesday

[burritojustice]

relevant js date functions, for reference

const unixTime = new Date(1580760378567);
const dateString = new Date('09/12/2016 12:00:00 AM');
const toUnix = Date.parse('09/12/2016 12:00:00 AM');

console.log('unixTime',unixTime);
console.log('datestring',dateString);

console.log('unixTime.toString',unixTime.toString());
console.log('unixTime.toISOString',unixTime.toISOString());

console.log('dateString.toString',dateString.toString());
console.log('dateString.toISOString',dateString.toISOString());
console.log('toUnix',toUnix);

this yields:

> "unixTime" Mon Feb 03 2020 12:06:18 GMT-0800 (Pacific Standard Time)
> "datestring" Mon Sep 12 2016 00:00:00 GMT-0700 (Pacific Daylight Time)
> "unixTime.toString" "Mon Feb 03 2020 12:06:18 GMT-0800 (Pacific Standard Time)"
> "unixTime.toISOString" "2020-02-03T20:06:18.567Z"
> "dateString.toString" "Mon Sep 12 2016 00:00:00 GMT-0700 (Pacific Daylight Time)"
> "dateString.toISOString" "2016-09-12T07:00:00.000Z"
> "toUnix" 1473663600000

[burritojustice]

node bin/here.js xyz upload 9AIVZHf2 -f https://eonet.sci.gsfc.nasa.gov/api/v3/events/geojson --date date

[burritojustice]

fixes

• add date_ prefix to all date tags we generate, before the property name
• Add leading zeros to month and day
• Use 01 for January instead of 0
• Use 01 for the first week of the year instead of 0
• Use week instead of week_of_year

timezones

• If there a timezone offset in an ISO 8601 string, preserve it, and use it for the tags
• Add a –timezone option to allow a user to designate a local timezone offset --timezone UTC+05:30 or --timezone UTC-09:00

time tag options

If a user does not specify any options after --datetag, we generate all the tags But let them choose a subset – list includes year, month, week, day, year_month, year_week, month_week

add an hour tag, 00-24 – …hour@10, …hour@18 – if we have a timezone offset, use it

Keep track of min / max dates and write to the user block of the space ID description?

[burritojustice]

added in 1.4, enhanced in 1.5

[burritojustice]

in some datasets, the date and time are separated — we should consider a + operator to pass two fields together for --date timestamping:

--date acq_date+acq_time