victorialogs-datasource icon indicating copy to clipboard operation
victorialogs-datasource copied to clipboard

Implement Grafana Query Builder

Open archef2000 opened this issue 6 months ago • 79 comments

Add support for all operations of LogsQL in the Grafana Query Builder. @Loori-R

#48

archef2000 avatar Jul 04 '25 13:07 archef2000

I am currently on 12.0.2 and everything is working will test version 11

archef2000 avatar Jul 07 '25 17:07 archef2000

The problem exists up to including v 11.4.6 and is fixed in 11.5.0. The source problem is that the custom Field Editors get undefined as props

archef2000 avatar Jul 07 '25 19:07 archef2000

It looks like the issue is with the Combobox component - it doesn't work correctly with older versions of Grafana. The Select component (which was its alternative) is now considered deprecated.

I think we should consider raising the minimum supported Grafana version. What do you think, @dmitryk-dk, @hagen1778?

Loori-R avatar Jul 08 '25 12:07 Loori-R

@archef2000, Could you please take a look at the tests and fix them? Here’s the link to the failing workflow: https://github.com/VictoriaMetrics/victorialogs-datasource/actions/runs/16075336610/job/45477811316?pr=329

Loori-R avatar Jul 08 '25 12:07 Loori-R

It looks like the issue is with the Combobox component - it doesn't work correctly with older versions of Grafana. The Select component (which was its alternative) is now considered deprecated.

I think we should consider raising the minimum supported Grafana version. What do you think, @dk, @roman?

In my experience query builders are fairly hard to use manually, - click click click hundred times, - I prefer plain old text for that. But what do I know, I am probably a wrong dk :)

dk avatar Jul 08 '25 12:07 dk

In my experience query builders are fairly hard to use manually, - click click click hundred times, - I prefer plain old text for that. But what do I know, I am probably a wrong dk :)

Oops, looks like I tagged the wrong person 😅 Thanks a lot for sharing your thoughts - and sorry for the ping!

Loori-R avatar Jul 08 '25 13:07 Loori-R

It looks like the issue is with the Combobox component - it doesn't work correctly with older versions of Grafana. The Select component (which was its alternative) is now considered deprecated.

I think we should consider raising the minimum supported Grafana version. What do you think, @dmitryk-dk, @hagen1778?

As for me, to up the version is ok, but we need to test everything properly

We could have another option to use, use the old Combobox, but I am not sure it would work in the newest versions.

11.3.1 as @Loori-R mentioned, is not an old version and many users are using versions 10-11.

dmitryk-dk avatar Jul 08 '25 13:07 dmitryk-dk

  • There is no "old" Combobox - it was introduced as a new component and is currently marked as Alpha. View source

  • The Select component is marked as Deprecated. View source

As an option, we could use Select for now since it’s more stable, and plan to migrate to Combobox in the future. Guide: Migrating from Select to Combobox

Loori-R avatar Jul 08 '25 13:07 Loori-R

  • There is no "old" Combobox - it was introduced as a new component and is currently marked as Alpha. View source
  • The Select component is marked as Deprecated. View source

As an option, we could use Select for now since it’s more stable, and plan to migrate to Combobox in the future. Guide: Migrating from Select to Combobox

I think using Select instead of Combobox is a better idea. We do not need to change the supported version, and we do not need to use the Alpha component.

dmitryk-dk avatar Jul 08 '25 13:07 dmitryk-dk

@Loori-R Can comfirm that with the Select component everything works.

archef2000 avatar Jul 08 '25 14:07 archef2000

@Loori-R Can comfirm that with the Select component everything works.

@archef2000 Would you mind replacing Combobox with Select?

Loori-R avatar Jul 08 '25 14:07 Loori-R

Already on it

archef2000 avatar Jul 08 '25 14:07 archef2000

Already on it

It looks like there are still some Combobox components in the code: https://github.com/VictoriaMetrics/victorialogs-datasource/pull/329/files#diff-35151418f87b2fad4cd2f8e9b4515cd9685e1c922d771a348463efc8672b15ecR12-R22

Loori-R avatar Jul 08 '25 14:07 Loori-R

I am not finished yet and will make a commit once all are changed and tested

archef2000 avatar Jul 08 '25 14:07 archef2000

This query should give me the value types of ClientPort right? /select/logsql/field_values?query=_msg%3A*+%7C+uniq+by+ClientPort+%7C+block_stats+%7C+uniq+type&start=1751839200000&end=1752011999999&field=type

query: _msg:* | uniq by ClientPort | block_stats | uniq type field: type

archef2000 avatar Jul 08 '25 15:07 archef2000

This query should give me the value types of ClientPort right? /select/logsql/field_values?query=_msg%3A*+%7C+uniq+by+ClientPort+%7C+block_stats+%7C+uniq+type&start=1751839200000&end=1752011999999&field=type

query: _msg:* | uniq by ClientPort | block_stats | uniq type field: type

Not sure, but it seems like uniq by ClientPort and uniq type might overlap. @dmitryk-dk, could you please help clarify?

Loori-R avatar Jul 09 '25 10:07 Loori-R

Seems like the query won’t return ClientPort types — it just gets unique type values from logs matching the full query as a text filter, because /field_values doesn’t run the pipeline logic.

Loori-R avatar Jul 09 '25 10:07 Loori-R

_msg:* | uniq by ClientPort | block_stats | uniq type

Sorry guys missed you question

query _msg:* | uniq by ClientPort | block_stats will return

https://play-vmlogs.victoriametrics.com/select/vmui/#/?query=_msg%3A*+%7C+uniq+by+ClientPort+%7C+block_stats&g0.range_input=5m&g0.end_input=2025-07-09T11%3A56%3A01&g0.relative_time=last_5_minutes

field: ClientPorttype: constvalues_bytes: 0bloom_bytes: 0dict_items: 0dict_bytes: 0rows: 1part_path: inmemory

the next filter will get only type so the query like _msg:* | uniq by ClientPort | block_stats | uniq type will return only type: const https://play-vmlogs.victoriametrics.com/select/vmui/#/?query=_msg%3A*+%7C+uniq+by+ClientPort+%7C+block_stats+%7C+uniq+type&g0.range_input=5m&g0.end_input=2025-07-09T11%3A55%3A21&g0.relative_time=last_5_minutes

dmitryk-dk avatar Jul 09 '25 11:07 dmitryk-dk

I was in the assumption that /field_values will return the values of the field that are available after the query. Just noticed it in the value_type operation, but then I will have to change the logic for all. Should I change the getFieldList function with an additional type?

archef2000 avatar Jul 09 '25 12:07 archef2000

I don't quite understand what you're trying to achieve.

For the expression: _msg:* | uniq by ClientPort | block_stats | uniq type

/fields_values can only be used for _msg (I'm not sure how usable it is for _msg, but still). To substitute values for uniq, you need to use /field_names

Loori-R avatar Jul 09 '25 12:07 Loori-R

For the value_type operation I want to get the availabe value types of the field specified. The field_values is explained so that I get all field values of the specified field from the result of the query. The query gives me the correct result, but the endpoint seems to ignore the query.

archef2000 avatar Jul 09 '25 13:07 archef2000

Maybe we could just hardcode the supported value types (like const, dict, string, int64, float64, etc.) and allow users to enter a custom type if needed - that might be a simpler and more flexible solution.

Loori-R avatar Jul 09 '25 14:07 Loori-R

I think we should consider raising the minimum supported Grafana version.

I don't like it. According to plugin stats, 50% of users are using <v12 Grafana version:

image

hagen1778 avatar Jul 15 '25 10:07 hagen1778

I think we should consider raising the minimum supported Grafana version.

I don't like it. According to plugin stats, 50% of users are using <v12 Grafana version:

image

We decided to use the old Select component and stay with the version we have now

dmitryk-dk avatar Jul 15 '25 10:07 dmitryk-dk

@dmitryk-dk

I suggest a little cleanup of the PR with callbacks, because every component has a handler, but its usage is on the element.

May you be able to give examples.

archef2000 avatar Jul 16 '25 11:07 archef2000

@dmitryk-dk

I suggest a little cleanup of the PR with callbacks, because every component has a handler, but its usage is on the element.

May you be able to give examples.

import React, { useState } from 'react';

import { SelectableValue } from '@grafana/data';
import { QueryBuilderOperationParamEditorProps, toOption } from '@grafana/plugin-ui';
import { InlineField, Select } from '@grafana/ui';

import { getFieldNameOptions } from './utils/editorHelper';

export default function FieldEditor(props: QueryBuilderOperationParamEditorProps) {
  const { value, onChange, index } = props;
  const [isLoading, setIsLoading] = useState(false);
  const [options, setOptions] = useState<SelectableValue<string>[]>([]);
  
  const onOpenMenu = async () => {
          setIsLoading(true);
          setOptions(await getFieldNameOptions(props));
          setIsLoading(false);
  }

 const handleOnChange = ({ value = "" }) => {
          onChange(index, value);
   }

  return (
    <InlineField>
      <Select<string>
        allowCustomValue={true}
        allowCreateWhileLoading={true}
        isLoading={isLoading}
        onOpenMenu={onOpenMenu}
        options={options}
        onChange={handleOnChange}
        value={toOption(String(value || ""))}
        width="auto"
      />
    </InlineField>
  );
}

Something like this

dmitryk-dk avatar Jul 16 '25 11:07 dmitryk-dk

Could the onChange not just be in one line? Like I did in ExactValueEditor.tsx

And just make handlers for onOpenMenu callbacks?

archef2000 avatar Jul 16 '25 11:07 archef2000

Could the onChange not just be in one line? Like I did in ExactValueEditor.tsx

And just make handlers for onOpenMenu callbacks?

I described just an example and asked to move all the handlers from the component to the handlers. It is better for the readability of the code. Please check all the components you provided, please. And check please if else return statement, because i found two places where else is redundant. It is nit a comments, but better to make it cleaner. :)

dmitryk-dk avatar Jul 16 '25 11:07 dmitryk-dk

I described just an example and asked to move all the handlers from the component to the handlers.

I would suggest extracting only complex logic - for example, if additional checks are needed or it takes more than 2-3 lines.

Loori-R avatar Jul 16 '25 13:07 Loori-R

Why are only filters respected in the field_values/field_names endpoints. When it is stated otherwise in the docs? @Loori-R @dmitryk-dk

archef2000 avatar Jul 18 '25 10:07 archef2000