langfuse icon indicating copy to clipboard operation
langfuse copied to clipboard

bug: Langfuse breaks llama index IngestionPipeline with parallelization

Open aiswaryasankar opened this issue 6 months ago • 2 comments

Describe the bug

The following code in llama index throws an error when run with langfuse

transformations = [
            HierarchicalNodeParser.from_defaults(chunk_sizes=[4096, 2048]),
            Settings.embed_model,
]
pipeline = IngestionPipeline(transformations=transformations)
nodes = pipeline.run(
    documents=createHierarchicalIndexRequest.Documents,
    num_workers=os.cpu_count(),
)```

The error thrown is `ERROR 2024-07-29 01:30:19,751 djangoMiddleware 4582 123145475784704  Exception occurred: cannot pickle '_thread.RLock' object Internal Server Error:` 

### To reproduce

Run the code above in llama index 

### SDK and container versions

_No response_

### Additional information

_No response_

### Are you interested to contribute a fix for this bug?

No

aiswaryasankar avatar Jul 31 '24 18:07 aiswaryasankar