Dynamic Indices

[PorterK]

Hi! I was wondering if you plan on adding (or if there is a way currently) dynamic indices. Our use case is for basic user logging/action tracking. We expect to be around 20k-30k new actions/day per client in our system. It would be really nice to break actions out into at least 1 index per client for now, but potentially 1 index/day/client in the future as the system grows.

Thank you!

[toluaina]

This is an interesting idea and I will need to give it more thought. As it stands, the replication slot is bound to the database and elastic search index. With a dynamic index, we need to decouple this somehow. There is also a one-off initialization that happens on both database and optionally elasticsearch index. I will explore this a bit further

[PorterK]

Thank you! Would love to help out in any way I can - I don't have a ton of knowledge around the cool SQL tricks that you are using under the hood, but if I can provide anything let me know. As far as the schema goes it would be cool to see something like this, I don't think it strays too far away from the setup you have now:

[
  {
    "database": "stuffraiser",
    "index": "org-{schema.organizationId}-actions",
    "dynamicIndex": true,
    "nodes": [
      {
        "table": "actions",
        "columns": [
          "id",
          "name",
          "organizationId"
        ],
        "children": [
          {
            "table": "users",
            "columns": [
              "id",
              "name",
              "username"
            ],
            "label": "user",
            "relationship": {
              "variant": "object",
              "type": "one_to_one"
            }
          }
        ]
      }
    ]
  }
]

[gabrielpra1]

I'm also interested in this feature! My use case is for storing a big amount of time-series data in Elasticsearch, so having separate indices by timeframe (e.g. one index per day) would be ideal.