ekuiper icon indicating copy to clipboard operation
ekuiper copied to clipboard

How to set the indexValue to the latest value?

Open Forstwith opened this issue 1 year ago • 3 comments

Now I have a source sql stream S1 , and I set the indexValue 1 and indexField id, and I have a rule R1 to log the message from S1.

When I first time to start R1, everything is fine, the R1 logged to indexValue 10 . But when I restart the R1 , the S1 start a new instance, and query start indexValue 1 again, which I want start from indexValue 10.

I read the official documentation, but I didn't find a solution.

Forstwith avatar Feb 06 '24 09:02 Forstwith

The indexValue is a state. It will be persisted if rule qos is set as below example. There are still some limitations, for detail please check https://ekuiper.org/docs/en/latest/guide/rules/state_and_fault_tolerance.html#enable-checkpointing

{
  "id": "ruleReadDB",
  "sql": "SELECT * from sqlDemo",
  "actions": [
    {
      "log": {
      }
    }
  ],
  "options": {
    "qos": 1,
    "checkpointInterval": 60000
  }
}

ngjaying avatar Feb 06 '24 11:02 ngjaying

@ngjaying Thank you very much for your answer, it's exactly what I needed.

But after I set qos=1 and checkPointInterval=60000, when I restart the rule(rule name is 0000002), I get the error.

error: closing rule 0000002 for error: topo 0000002 create store error SQL logic error: near "0000002": syntax error (1)" file="rule/ruleState.go:214

I looked at the source code and it seems that an error occurred when executing the open method.

Here are the logs.

time="2024-02-07T02:44:59Z" level=info msg="Init rule with options &{Debug:false LogFilename: IsEventTime:false LateTol:1000 Concurrency:1 BufferLength:1024 SendMetaToSink:false SendError:true Qos:1 CheckpointInterval:60000 Restart:0x4000ac8b70 Cron: Duration: CronDatetimeRange:[]}" file="planner/planner.go:44"

time="2024-02-07T02:44:59Z" level=info msg="Opening stream" file="topo/topo.go:183" rule=0000002

time="2024-02-07T02:44:59Z" level=error msg="runtime error: topo 0000002 create store error SQL logic error: near "0000002": syntax error (1)" file="infra/saferun.go:56" rule=0000002

time="2024-02-07T02:44:59Z" level=error msg="closing rule 0000002 for error: topo 0000002 create store error SQL logic error: near "0000002": syntax error (1)" file="rule/ruleState.go:214" `

More info:

  1. ekuiper version: 1.12.3-alpine
  2. Run using docker container.

Forstwith avatar Feb 18 '24 06:02 Forstwith

Could you try another rule id like rule000002?

ngjaying avatar Feb 19 '24 02:02 ngjaying

@ngjaying After change rule id to rule000002, works fine. Thank you very much.

Do you think this is a bug? Because the 0000002 rule id will cause failure to start rule when the Qos setting is not 0

Forstwith avatar Feb 20 '24 02:02 Forstwith

Kind of, but we will recommend to not set id to start with number. We are adding rule id validation.

ngjaying avatar Feb 20 '24 03:02 ngjaying

Okay, then this issue can be closed. Again, thank you very much for your help.

Forstwith avatar Feb 20 '24 03:02 Forstwith