siddhi icon indicating copy to clipboard operation
siddhi copied to clipboard

How to write a Kryo serializer for SiddhiAppRuntime?

Open salvalcantara opened this issue 5 years ago • 3 comments

As a first attempt, I wrote the following (non-working) code:

package com.my.alerting.siddhi

import com.esotericsoftware.kryo.io.{Input, Output}
import com.esotericsoftware.kryo.{Kryo, Serializer}
import io.siddhi.core.SiddhiAppRuntime

class SiddhiAppRuntimeSerializer extends Serializer[SiddhiAppRuntime] {
  override def write(kryo: Kryo, output: Output, runtime: SiddhiAppRuntime): Unit = {
    val runtimeSnapshot = runtime.snapshot()
    output.writeInt(runtimeSnapshot.length)
    output.writeBytes(runtimeSnapshot)
  }

  override def read(kryo: Kryo, input: Input, `type`: Class[SiddhiAppRuntime]): SiddhiAppRuntime = {
    val runtimeSnapshotLength = input.readInt()
    val runtimeSnapshot = input.readBytes(runtimeSnapshotLength)
    val runtime = `type`.newInstance()
    runtime.restore(runtimeSnapshot)

    runtime
  }
}

But I realize that I cannot create a new runtime instance like this val runtime = type.newInstance(), so I am trying something else:

override def write(kryo: Kryo, output: Output, runtime: SiddhiAppRuntime): Unit = {
  val app = runtime.getSiddhiApp.toString
  val runtimeSnapshot = runtime.snapshot()
  output.writeString(app)
  output.writeInt(runtimeSnapshot.length)
  output.writeBytes(runtimeSnapshot)
 }
override def read(kryo: Kryo, input: Input, `type`: Class[SiddhiAppRuntime]): SiddhiAppRuntime = {
  val app = input.readString()
  val runtimeSnapshotLength = input.readInt()
  val runtimeSnapshot = input.readBytes(runtimeSnapshotLength)
  val manager = new SiddhiManager()
  val runtime = manager.createSiddhiAppRuntime(app)
  runtime.restore(runtimeSnapshot)

  runtime
 }

However the above approach also fails because runtime.getSiddhiApp.toString does not return the whole Siddhi App code as I was expecting, so I cannot use that string for creating a SiddhiAppRuntime using the manager. Another concern is having to create a SiddhiManager within the read method...which I would prefer to avoid if possible.

Please, can anyone help on this?

salvalcantara avatar Mar 04 '20 15:03 salvalcantara

FYI, I posted the same question in https://stackoverflow.com/questions/60529065/how-to-write-a-kryo-serializer-for-siddhiappruntime.

salvalcantara avatar Mar 05 '20 07:03 salvalcantara

Also, as an aside note, I think it should be possible to recover the full siddhi app code (as a string) from the runtime object, what do you guys think @mohanvive ?

salvalcantara avatar Mar 05 '20 11:03 salvalcantara

As an update, this is the best thing I have come up with so far:

import com.esotericsoftware.kryo.io.{Input, Output}
import com.esotericsoftware.kryo.{Kryo, Serializer}
import io.siddhi.core.{SiddhiAppRuntime => SiddhiRuntime}

// A simple wrapper necessary because of https://github.com/siddhi-io/siddhi/issues/1630
case class SiddhiAppRuntime(app: String, runtime: SiddhiRuntime)

class SiddhiAppRuntimeSerializer() extends Serializer[SiddhiAppRuntime] with Serializable {

  override def write(kryo: Kryo, output: Output, appRuntime: SiddhiAppRuntime): Unit = {
    val runtimeSnapshot = appRuntime.runtime.snapshot()

    output.writeString(appRuntime.app)
    output.writeInt(runtimeSnapshot.length)
    output.writeBytes(runtimeSnapshot)
  }

  override def read(kryo: Kryo, input: Input, `type`: Class[SiddhiAppRuntime]): SiddhiAppRuntime = {
    val app                   = input.readString()
    val runtimeSnapshotLength = input.readInt()
    val runtimeSnapshot       = input.readBytes(runtimeSnapshotLength)

    // Isn't there a way to avoid this??
    val siddhiManager = new SiddhiManager()

    val runtime = siddhiManager.createSiddhiAppRuntime(app)
    runtime.restore(runtimeSnapshot)

    SiddhiAppRuntime(app, runtime)
  }
}

The main problem with this approach is that I rely on a SiddhiManager object in order to be able to recreate the runtime from a string representation of the Siddhi app, and before restoring the snapshot.

How could I avoid having to create a SiddhiManager in the read method? I guess I could rely on a global singleton object since, indeed, Siddhi seems to recommend using one SiddhiManager per JVM. However, I wonder whether this could result in contention issues (imagine a lot of threads using different runtimes).

salvalcantara avatar Apr 08 '20 09:04 salvalcantara