[Issue]: Can we use open source Large models as agent ?

[whatUwilldoknowingmyname]

Describe the issue

This question is general, not specific to text. Can I use any other i.e diffusion model or any other model as agent ? And use this all agents, llm and diffusion model, in the form of multi-agent so that, they can interact with each other.

Steps to reproduce

No response

Screenshots and logs

No response

Additional Information

No response

[rickyloynd-microsoft]

@BeibinLi fyi

[BeibinLi]

@whatUwilldoknowingmyname Yes, check

• LLaVA: https://github.com/microsoft/autogen/blob/main/notebook/agentchat_lmm_llava.ipynb
• DALLE: https://github.com/microsoft/autogen/blob/main/notebook/agentchat_dalle_and_gpt4v.ipynb

Even if DALLE is not open source, using other open-source stable diffusion model would be the same process.