stats by do not support date_format function

[chyun]

I input PPL "search source=210324|stats max(event_duration) by date_format(inbuilt_meta.log_timestamp, 'yyyy-MM-dd HH')". Got the error { "error": { "reason": "Invalid Query", "details": "Failed to parse query due to offending symbol [date_format] at: 'search source=*210324*|stats max(event_duration) by date_format' <--- HERE... More details: Expecting tokens in {'D', '.', '', 'AVG', 'COUNT', 'MAX', 'MIN', 'SUM', 'FIRST', 'LAST', 'DATE', 'TIME', 'TIMESTAMP', ID, BQUOTA_STRING}", "type": "SyntaxCheckException" }, "status": 400 }`

It looks like stats by do not support very much function

[chyun]

I can use "search source=210324|fields event_duration, inbuilt_meta.log_timestamp|eval day_str=date_format(inbuilt_meta.log_timestamp, 'yyyy-MM-dd HH')" to get the right answer, but the method do not aggregate all of data, it just aggregate the head part of data

[dai-chen]

Thanks for reporting the issue! Investigating

[dai-chen]

It seems BY clause only accepts field name: https://github.com/opendistro-for-elasticsearch/sql/blob/develop/ppl/src/main/antlr/OpenDistroPPLParser.g4#L107. We need to check if this can be enabled easily. For now, could you try to append stats to your eval command. Note that this may have performance issue on big data set. Thanks!