Use @timestamp instead of timestamp.us and switch to type date_nanos

[simitt]

Switch to @timestamp from timestamp.us:

• drop writing to timestamp.us in favor of @timestamp
• change @timestamp type to date_nanos with a default format of epoch_millis
• add a runtime field for timestamp.us until we can move fully onto @timestamp.

The UI uses the fields API to query @timestamp with format strict_date_optional_time_nanos, and also queries timestamp.us and uses that if it exists (https://github.com/elastic/kibana/issues/108908). We would stop mapping and indexing the latter, this would just be for backwards compatibility.

There are known issues in Kibana with supporting date_nanos types, which needs further attention before we can move forward with this change.

---

Known issues that require further discussions:

• Kibana (javascript) issues supporting date_nanos https://github.com/elastic/kibana/issues/40183
• Impact on logs UI https://github.com/elastic/kibana/issues/88290

Related issues:

• Kibana APM UI changes https://github.com/elastic/kibana/issues/108908
• beats changes for @timestamp precision https://github.com/elastic/beats/issues/15871

[jalvz]

we are not going do this now, right?

[jalvz]

we can reopen by 8.0 timeframe

[simitt]

Reopening and targeting 8.0; updated the description to reflect current status and plans.

[simitt]

Next steps:

• [ ] track down required alignment on @timestamp values for logs UI (ECS specifies type date https://www.elastic.co/guide/en/ecs/current/ecs-base.html#field-timestamp) and impact when changing to date_nanos

[axw]

The APM app itself appears to have no problems with date_nanos. I installed the integration, edited the index templates (changing date to date_nanos for @timestamp), and modified https://github.com/elastic/apm-server/pull/5970 to encode events with nanosecond precision.

I then sent some events and the APM app does not appear to have any problems with this.

I've found a couple of issues so far:

• package-registry disallows use of date_nanos for @timestamp. Probably not hard to change.
• Given that we write errors to a logs data stream, changing to date_nanos there triggers https://github.com/elastic/kibana/issues/88290. We could use date for logs, but our errors also have microsecond resolution.

What we could do to work around this (temporarily, until the Logs app is fixed) is:

• use date_nanos for traces data streams
• use date for @timestamp in logs data streams, and index a sub-millisecond timestamp offset separately

For trace events, I think we'll probably need a runtime field for accessing sub-millisecond precision, similar to the example in https://www.elastic.co/guide/en/elasticsearch/reference/master/date_nanos.html

[axw]

For trace events, I think we'll probably need a runtime field for accessing sub-millisecond precision, similar to the example in https://www.elastic.co/guide/en/elasticsearch/reference/master/date_nanos.html

Given that we'll need this anyway, I guess there's not much point in removing timestamp.us.

My new proposal is:

• update timestamp.us to be non-indexed; it is never queried AFAICS
• keep @timestamp as date for now, until ECS officially supports @timestamp being date_nanos (https://github.com/elastic/ecs/issues/1065) and https://github.com/elastic/kibana/issues/88290 is fixed
• when we move @timestamp to date_nanos, update timestamp.us to be a runtime field, possibly defined in the Data View

I'll run some more tests to check that this works out.

[axw]

Found a bug in Kibana/Fleet: https://github.com/elastic/kibana/issues/109764

when we move @timestamp to date_nanos, update timestamp.us to be a runtime field, possibly defined in the Data View

I believe the UI is currently extracting all fields from _source, rather than using the Fields API. We would need to migrate to the Fields API to make use of runtime fields: https://github.com/elastic/kibana/issues/84507

[axw]

Another option occurs to me now: we can continue to store @timestamp as a date field, but encode it using strict_date_optional_time_nanos. The UI can pull that out of _source and parse it for ordering things on the waterfall.