[Feature Request] Add DSP to langchain

[atemaguer]

DSP is a framework that provides abstractions and primitives for easily composing language and retrieval models. Furthermore, it proposes an algorithmic approach to prompting LLMs that does away with prompt engineering

https://arxiv.org/pdf/2212.14024.pdf, https://github.com/stanfordnlp/dsp

[atemaguer]

@hwchase17, I'm open to working on this if we can agree on the right API for langchain

[conceptofmind]

@atemaguer We have a group on the discord currently working on adding DSP to LangChain: https://discord.com/channels/1038097195422978059/1068175360648286248

[atemaguer]

The discord link you provided doesn't seem to be valid

[conceptofmind]

The discord link you provided doesn't seem to be valid

Hmmm. It worked when I tried it just now. Is there a discord name I can just @ you in the chat with?

[aabdullah-getguru]

@conceptofmind hello! I'm squared_circle on discord. I don't know what server and channel you're referring to, it's probably a private server :). Out of curiosity, what is the status of this work? Like atemaguer , I'd be interested in adding this if someone else isn't working on it.

[conceptofmind]

@conceptofmind hello! I'm squared_circle on discord. I don't know what server and channel you're referring to, it's probably a private server :). Out of curiosity, what is the status of this work? Like atemaguer , I'd be interested in adding this if someone else isn't working on it.

Hi,

We already have a large majority of it done. I can add you to the thread.

Best,

Enrico

[aabdullah-getguru]

@conceptofmind hello! I'm squared_circle on discord. I don't know what server and channel you're referring to, it's probably a private server :). Out of curiosity, what is the status of this work? Like atemaguer , I'd be interested in adding this if someone else isn't working on it.

Hi,

We already have a large majority of it done. I can add you to the thread.

Best,

Enrico

Hey Enrico. Thanks that would be wonderful if you can! I'd appreciate it :).

[conceptofmind]

@conceptofmind hello! I'm squared_circle on discord. I don't know what server and channel you're referring to, it's probably a private server :). Out of curiosity, what is the status of this work? Like atemaguer , I'd be interested in adding this if someone else isn't working on it.

Hi, We already have a large majority of it done. I can add you to the thread. Best, Enrico

Hey Enrico. Thanks that would be wonderful if you can! I'd appreciate it :).

What is the four digit tag after your username?

[aabdullah-getguru]

@conceptofmind hello! I'm squared_circle on discord. I don't know what server and channel you're referring to, it's probably a private server :). Out of curiosity, what is the status of this work? Like atemaguer , I'd be interested in adding this if someone else isn't working on it.

Hi, We already have a large majority of it done. I can add you to the thread. Best, Enrico

Hey Enrico. Thanks that would be wonderful if you can! I'd appreciate it :).

What is the four digit tag after your username?

Whoops. 9344. Thanks!

[darien-schettler]

Hi there, I’m just following up on this.

Is there progress related to DSP integration? Or at least an open pull request.

I’m hoping to understand if it’s worth investing my own time from scratch, contributing to an existing branch, or just waiting/using if it’s done.

Thanks in advance!

[aabdullah-getguru]

Hi there, I’m just following up on this.

Is there progress related to DSP integration? Or at least an open pull request.

I’m hoping to understand if it’s worth investing my own time from scratch, contributing to an existing branch, or just waiting/using if it’s done.

Thanks in advance!

Not sure either. There seems to be this repo here which has a lot of this implemented in langchain, but its been silent a couple of months: https://github.com/conceptofmind/dsp-langchain

[conceptofmind]

Hi there, I’m just following up on this.

Is there progress related to DSP integration? Or at least an open pull request.

I’m hoping to understand if it’s worth investing my own time from scratch, contributing to an existing branch, or just waiting/using if it’s done.

Thanks in advance!

As @aabdullah-getguru has noted. We have already implemented most of DSP. There was a small part of program 5 we had not added yet. If you can get Omar's approval, I can just open a PR with what I have already done.

It is an interesting academic research avenue though.

Also, the main author of DSP does not want it added to LangChain and has publicly stated so. So, I guess depends if you want to respect his wishes as well.

[michox]

I thought the point of DSP was that once you have built the tools you can now use much smaller models like Ada or a local model and still achieve a similar performance? see here for example https://github.com/stanfordnlp/dsp/blob/main/compiler.ipynb

[dosubot[bot]]

Hi, @atemaguer! I'm Dosu, and I'm helping the LangChain team manage their backlog. I wanted to let you know that we are marking this issue as stale.

From what I understand, this issue is a feature request proposing the addition of DSP (Deep Semantic Parsing) to LangChain. There has been a discussion in the comments about this, with @conceptofmind mentioning that a group on Discord is already working on it and has made significant progress. However, @conceptofmind also mentioned that the main author of DSP does not want it added to LangChain.

Before we close this issue, we wanted to check with you if it is still relevant to the latest version of the LangChain repository. If it is, please let us know by commenting on the issue. Otherwise, feel free to close the issue yourself or it will be automatically closed in 7 days.

Thank you for your understanding!