bug: genai plugin unable to share context between multiple instances of Backstage

[viniagostini]

📜 Description

Hello team! I am experimenting with the genai plugins, and it's been great! Really powerful stuff.

I am running backstage with more than one instance for HA reasons, and since the chat persistence is in memory, as I can see from here, and therefore two different instances can't share the context of the conversation. With this, my conversations with the AI Assistant restart whenever the request is forwarded to a new instance.

👍 Expected behavior

I see on the documentation that not having persistence is listed as a known limitation of this implementation. What I am seeking here is to understand if you have plans to tackle this issue or if you have suggestions on how to workaround this problem on my side.

👎 Actual Behavior with Screenshots

👟 Reproduction steps

Install the plugin and provide more than one backend instance.

📃 Content

No response

🖥️ Your Environment

No response

[niallthomson]

Hey @viniagostini ! Glad to hear you're finding the plugin interesting.

What you've brought here is less of a bug and more of a feature gap thats called out in the README: currently we only use in-memory persistence for chat history.

To fill this gap we'd need to provide an option to use something like the Postgres persistence in LangGraph, which isn't something I've had time to fully explore. Ideally we could find an implementation that re-uses the Postgres database of Backstage to store this information so it doesn't add much complexity.

[viniagostini]

What you've brought here is less of a bug and more of a feature gap thats called out in the README: currently we only use in-memory persistence for chat history.

Totally agree, sorry to tag this as a bug

To fill this gap we'd need to provide an option to use something like the Postgres persistence in LangGraph, which isn't something I've had time to fully explore. Ideally we could find an implementation that re-uses the Postgres database of Backstage to store this information so it doesn't add much complexity.

This is one of the options I have considered: Creating a custom agent implementation that stores the Langraph context on the plugin's DB. I'll explore this path, and if it works well and you're open to contributions, I can move my implementation to here 😄

BTW, I really like how this plugin leverages extension points to enable these kinds of customization. Congrats!

[niallthomson]

If you're able to get the Postgres path working I'd be very interested in seeing it! I also looked in to something a bit more "lightweight" for AWS users like DynamoDB but that would have been a lot of work I didn't have bandwidth for.

[viniagostini]

I was able to get it working with Postgres, but not in a very Backstage Native way, so I am not sure if you would want to incorporate it into this plugin

Basically, I have written my own Agent using the extension point as I mentioned. The core changes on the Agent implementation are:

[...]
import { PostgresSaver } from '@langchain/langgraph-checkpoint-postgres';
[...]

    let checkpointSaver: BaseCheckpointSaver

    const dbConfig = rootConfig.getConfig('backend.database');
    const dbClient = dbConfig.getString('client');
    if (dbClient === 'pg') {
      const dbConnection = dbConfig.getConfig('connection');
      const host = dbConnection.getString('host');
      const port = dbConnection.getNumber('port');
      const user = dbConnection.getString('user');
      const password = dbConnection.getString('password');
      const connectionString = `postgresql://${user}:${password}@${host}:${port}/backstage_plugin_aws-genai`;
      const postgresSaver = PostgresSaver.fromConnString(connectionString);
      await postgresSaver.setup();
      checkpointSaver = postgresSaver;
    } else {
      checkpointSaver = new MemorySaver();
    }

    const agent = createReactAgent({
      llm: agentModel,
      tools,
      checkpointSaver,
      [...]
    });

As you can see, this approach doesn't reuse the Backstage database connection pool and doesn't account for the full range of possible database configuration scenarios. I'm not sure how much effort it would take to make it generic enough to support all use cases, but it has worked well for me. I hope this is helpful for you and others facing a similar challenge.

Here is it working with more than one replica

[secustor]

@viniagostini

Have you tried this?

static async from(database DatabaseService) {
            const client = await database.getClient();

            const conconfig = client.client.config.connection

            const { user, password, host, port, database } = connectionConfig;
            const connectionString = `postgresql://${user}:${password}@${host}:${port}/${database}`;
            const postgresSaver = PostgresSaver.fromConnString(connectionString);
}

This uses the Backstage plugin interface to fetch the connection details. This assumes there will be no conflicts with the table names.

[niallthomson]

I did some experimentation last week and was able to get this working pretty much as you outlined @secustor Its able to use either the sqlite or postgres database depending how you configured Backstage.

I will try to get this merged in the next week or so.