Philipp Schmid
Philipp Schmid
Which versions are you using?
and transformers/datasets?
Does using session.call_tool works, without any LLM inteaction?
Hey @sck, what GPU are you using and how big is your dataset? can you try decreasing the batch_size more, like to 8 or something
Hello Streaming is not yet support on SageMaker, easyllm is not using any layer in between. I hope they ll add support soon then i ll add it as well.
Can you check out this example for sagemaker? and see if it works? https://philschmid.github.io/easyllm/examples/sagemaker-chat-completion-api/#1-import-the-easyllm-library