🚀 Feature: Support DSPy

[nirga]

Which component is this feature for?

All Packages

🔖 Feature description

Instrument DSPy framework

🎤 Why is this feature needed ?

✌️ How do you aim to achieve this?

Similar to our LlamaIndex & LangChain instrumentations

🔄️ Additional Information

No response

👀 Have you spent some time to check if this feature request has been raised before?

• [X] I checked and didn't find similar issue

Are you willing to submit PR?

None

[parthmshah1302]

@nirga I would like to contribute to this. I did check the contribution guidelines but am looking for more instructions on how to contribute - can you please help me?

[nirga]

@parthmshah1302 sorry for the delay here - would love to assist you! You're welcome to join our community slack and we can help guide you. Basically the first step would be to build a super-simple sample app in the sample app package. This app can basically be some adaptation of DSPy's getting started docs. Then, you can start working on the instrumentation. Copy one of the existing ones and work your way. You need to report traces that show runs of DSPy - this will require looking at the code to understand for example what happens when you call Evaluate.__call__

[someshfengde]

Hi @nirga Somesh here, can you assign this issue to me I'll start working on it asap

[nirga]

@someshfengde you got it!

[someshfengde]

Hi @nirga , I'm trying to figure out what exactly do I have to implement. Tldr; Evaluate.call will not capture all calls as some calls were occured during program compilation, bootstrapping etc.

From what I've understood till now we have to create a wrapper so that it can capture LLM calls occurs during the dspy evaluate function call.

But evaluation method directly doesn't use LLM calls. calls were being made by program via dspy/predict/predict.py ( with help of already configured LM )

Adding wrapper over Evaluate will not capture all calls but adding over predict will capture all LLM calls.

Let me know what you think?

Thanks.

[nirga]

Hey @someshfengde! Sorry for the delay here as I was traveling. I see you're on our slack channel so I'll continue the discussion there!

[pranaybattu]

Hi @nirga , if no one else is working on this. Can you assign this to me!

[someshfengde]

Hi @pranaybattu I'm working on it but still facing complexities with wrapping dspy library. If you want to collaborate let me know. :)

[pranaybattu]

Sure @someshfengde, I'd love to collaborate. Just let me know where you're stuck, and we can dive into it together!

[someshfengde]

Are you available on slack? let's continue over there?