langfuse
langfuse copied to clipboard
bug: Langfuse breaks llama index IngestionPipeline with parallelization
Describe the bug
The following code in llama index throws an error when run with langfuse
transformations = [
HierarchicalNodeParser.from_defaults(chunk_sizes=[4096, 2048]),
Settings.embed_model,
]
pipeline = IngestionPipeline(transformations=transformations)
nodes = pipeline.run(
documents=createHierarchicalIndexRequest.Documents,
num_workers=os.cpu_count(),
)```
The error thrown is `ERROR 2024-07-29 01:30:19,751 djangoMiddleware 4582 123145475784704 Exception occurred: cannot pickle '_thread.RLock' object Internal Server Error:`
### To reproduce
Run the code above in llama index
### SDK and container versions
_No response_
### Additional information
_No response_
### Are you interested to contribute a fix for this bug?
No