InfluxQL shell doesn't work with auto-mapped DBRPs

[sanderson]

InfluxDB 2.4 will introduce DBRP mappings that get automatically created when you create a bucket using the db/rp naming convention. However, the InfluxQL shell doesn't recognize these mappings

Steps to reproduce:

1. Create a new bucket named hello/world.

2. Ensure the DBRP mapping was created:

   $ influx v1 dbrp ls
   ID                     Database                     Bucket ID              Retention Policy       Default     Virtual    Organization ID
   f11297a9efc0f42d       _monitoring                  f11297a9efc0f42d       autogen                false       true       a84158a5bb8a86e2
   a4068904a4ac71f7       _tasks                       a4068904a4ac71f7       autogen                false       true       a84158a5bb8a86e2
   ac0c7bbf26d8219a       hello                        ac0c7bbf26d8219a       world                  false       true       a84158a5bb8a86e2
   

3. Start an InfluxQL shell:

   influx v1 shell
   

4. Try to use the newly created DBRP mapping:

   InfluxQL Shell dev
   Connected to InfluxDB OSS dev
   > use hello
   No such retention policy "" exists on "hello"
   Available retention policies on "hello":
   > use hello.world
   No such retention policy "world" exists on "hello"
   Available retention policies on "hello":
   

Expected behavior: I expect to be able to use the auto-mapped DBRP combination in the InfluxQL shell.

Actual behavior: The shell recognizes the database but not the retention policy in the mapping

Environment info:

• System info: Darwin 20.6.0 x86_64
• InfluxDB version: InfluxDB dev (git: 78c969e510) build_date: 2022-08-10T21:43:15

[sanderson]

Other bugs that I've found:

• Virtual DBRP mappings do not have a default retention policy, so you have to explicitly specify the autogen RP. I think we should make autogen or the parsed RP the default RP for virtual DBRPs.
• SHOW MEASUREMENTS doesn't work with any virtual DBPRs, but it works with "hard" DBRPs.

[sanderson]

@candrewlee14

[sanderson]

Another issue I've found is that when you create multiple buckets with the same "db" name, SHOW RETENTION POLICIES doesn't return all the retention policies (in the InfluxQL shell). I think this is related to the other issue where it doesn't recognize all the retention policies associated with a db name:

To replicate:

1. Create 3 new buckets name test, test/foo, and test/bar.

2. Confirm that the virtual DBRP mappings were created.

   $ influx v1 dbrp ls
   ID                  Database       Bucket ID           Retention Policy    Default    Virtual    Organization ID
   f11297a9efc0f42d    _monitoring    f11297a9efc0f42d    autogen             false      true       a84158a5bb8a86e2
   a4068904a4ac71f7    _tasks         a4068904a4ac71f7    autogen             false      true       a84158a5bb8a86e2
   4c4aa84808f7776a    test           4c4aa84808f7776a    autogen             false      true       a84158a5bb8a86e2
   4d6ebfee271bfa69    test           4d6ebfee271bfa69    bar                 false      true       a84158a5bb8a86e2
   920782adf9faa77b    test           920782adf9faa77b    foo                 false      true       a84158a5bb8a86e2
   

3. Start and InfluxQL shell

   influx v1 shell
   

4. Use the test database and try to list retention policies:

   > use test
   > show retention policies
   name    duration shardGroupDuration replicaN default
   ----    -------- ------------------ -------- -------
   autogen 0s
   

I would expect the other two retention policies to show up in the output.

[sanderson]

This isn't unique to the InfluxQL shell. I get the same results with a raw API call:

$ curl --get http://localhost:8086/query?db=test \
  --header "Authorization: Token mytoken" \
  --data-urlencode "q=SHOW RETENTION POLICIES"

Returns:

{
  "results": [
    {
      "statement_id": 0,
      "series": [
        {
          "columns": [
            "name",
            "duration",
            "shardGroupDuration",
            "replicaN",
            "default"
          ],
          "values": [
            [
              "autogen",
              "0s",
              "168h0m0s",
              1,
              false
            ]
          ]
        }
      ]
    }
  ]
}

[candrewlee14]

I've found the problem, and it has to do with the way I handle reverse searching buckets from queries without database name and/or retention policy. For example, finding all buckets with the database name "hello" requires looking through all bucket names and parsing ones with "hello/foo". Working on a fix now, and adding more test cases.

[elyscape]

I still get Error: retention policy not found: autogen when I try to use the InfluxQL shell with an auto-mapped DBRP. This happens both with buckets that existed prior to the upgrade to InfluxDB 2.4.0 and bucket created after the upgrade.

[elyscape]

@candrewlee14 Making sure you see this since I can't reopen the issue. I figured this is preferable to opening a new issue.

[jeffreyssmith2nd]

Thanks for the report, I'll go ahead and reopen this to make it easier to track.

[Zincr0]

I still get Error: retention policy not found: autogen when I try to use the InfluxQL shell with an auto-mapped DBRP. This happens both with buckets that existed prior to the upgrade to InfluxDB 2.4.0 and bucket created after the upgrade.

Getting the same error. Is there a workaround to this or the compatibility layer to 1.x clients is currently broken?

[jeffreyssmith2nd]

Leaving the results of an offline discussion here:

The expected behaviour for dbrp mappings is:

• A user should be able to create a db/rp that points to a bucket (a "physical" mapping)
• For all buckets, we should create a bucket_name/autogen virtual mapping unless a physical mapping of the same db/rp exists.

Default behaviour:

• an explicit DBRP mapping can be marked as default, and that should be honored
• If there are no explicit mappings, but there is a bucket named dbname (not dbname/rpname) then that is the default
• Otherwise, give an error when a query would try to select a default - the error should say something nice like ‘please set up explicit dbrp mappings or set the rp explicitly’.

[davidby-influx]

@jeffreyssmith2nd - why would the default be dbname without an rp, instead of dbname/autogen?

[jeffreyssmith2nd]

The main use case is for people to be able to create a bucket and immediately be able to query it with InfluxQL but not have to worry about DBRPs if they choose not to (likely a new 2.x user that likes InfluxQL for the syntax rather than someone with historical ties to 1.x). So any bucket of just the bucket-name format has a virtual dbrp of bucket-name.autogen and is set to default to make it easier to jump right into querying.