CopilotKit
📚 Documentation: Aadd AWS Lambda example for self hosted copilotkitRuntime
💬 Let us know how we can improve our documentation.
Provide an example to make a copilotKitRuntime into an aws lambda behind an aws apigateway.
Hey @ClementVanPeuter, thanks for the request. I'd like to understand more context because I will create a customer need ticket. What is the level of impact without this example?
Hi @NathanTarbert. We love CopilotKit, and we already deploy a self hosted copilot runtime in the form of a node js express endpoint, successfully.
But it's not great, all of our services are in AWS ( in Lambda to be precise, EC2 instances are a no go for us, for politicals reasons).
Today I struggle to adapt my node js express code to a lambda code.
I've used the serverless-http package to make my code "compatible". And it's works fine until the handler code.
My lambda is behind an API GAteway like https://jhrtic3eh.execute-api.us-east-2.amazonaws.com/default/copilotkit-endpoint, and the copilotRuntimeNodeHttpEndpoint require an endpoint.
app.use('*', async (req, res, next) => {
console.log('Handling request post');
const handler = copilotRuntimeNodeHttpEndpoint({
endpoint: '/copilotkit-endpoint', <- THIS URL
runtime,
serviceAdapter: serviceAdapter,
});
console.log('Handler created');
const response = await handler(req, res, next);
console.log('Response received');
console.log("RESPONSE:",response);
return handler(req, res, next);
});
export const handler = serverless(app);
So in my logs i see the console logs but response is undefined. And I strongly believe it's due to the fact the handler endpoint it's not correct and colide in a way with the API GATEWAY settings.
So it's possible to have an AWS lambda ( with api gateway) example in the doc section in addition with the others ?
This greatly slows down adoption of copilotKit in our teams and in our products :/
Hey I am new to open-source, @ClementVanPeuter can you tell me which files should I be looking into, then I can take this up, thanks!
@ClementVanPeuter, we would like to get some more context here. Would you be open to a quick call?
@NathanTarbert , with pleasure, I've already booked a slot on Graham cal. is that ok ? Sorry for the time response I was on vacation.
To summarize, can we deploy an endpoint as documented here -> https://docs.copilotkit.ai/guides/self-hosting But in a serverless environment like AWS Lambda?
Hey @ClementVanPeuter, no worries at all.
Yes it's ok, I just didn't want to leave Graham's calendar link public.
Ariel, CopilotKit Dev is very familiar with AWS Lambda but there is a lack of support for streaming.
Do you have experience running an LLM workload in prod with AWS Lambda + API Gateway?
@arielweinberger, let me know he'd be happy to be your contact on this.
@NathanTarbert Perfect!
Currently we have successfully queried basic Bedrock models and a basic Bedrock agent in a traditional way via Lambda with an API gateway and a Lambda authorizer.
With copilotkit runtime and langchain adapter in a serverless env the code doesn't work and we don't know why. Yet, It does work in a classic express app.
Curious to find out if anyone has gotten this to work and can share an example. Thanks!
@srallen I just spent some time looking at this setup and my conclusion was that API Gateway or even just lambda for that matter will not work well with co-pilot kit as the chat updates will not "stream" to the client.
My understanding is that both API Gateway and Lambda require the full data stream to be buffered before sending a response.
@ChuckJonas, in general response streaming is possible with Lambda since 2023 (https://aws.amazon.com/blogs/compute/introducing-aws-lambda-response-streaming/).
Do you mean there is a limitation that has to do with how CopilotKit works, so that lambda response streaming cannot be used?
Lambda since 2023
Oh nice, I didn't realize they had added support. I've mostly used Lambdas behind AWS gateway (which still does't support streaming) or event bridge.
Do you mean there is a limitation that has to do with how CopilotKit works
I'm not aware of any limitation. It seems like it would work fine using the copilotRuntimeNodeHttpEndpoint.
I imagine you'd need to adapt the request/response object to match the lambda event format?