Nice elonmusk bot, but make him current: Please also support RAG for social media such as twitter

[yosun]

🚀 The feature

Hey, Since you have elonmusk as your main readme example, please have that example also include social media such as twitter.com and instagram.com (etc)

import os
from embedchain import Pipeline as App

# Create a bot instance
os.environ["OPENAI_API_KEY"] = "YOUR API KEY"
elon_bot = App()

# Embed online resources
elon_bot.add("https://en.wikipedia.org/wiki/Elon_Musk")
elon_bot.add("https://www.forbes.com/profile/elon-musk")

# Query the bot
elon_bot.query("What's the latest with Elon Musk?") 

  File "/Users/mbp16_2022_m1max/MyTests/python/test_embedchain.py", line 11, in <module>
    elon_bot.add("https://www.x.com/elonmusk")
  File "/Users/mbp16_2022_m1max/MyTests/python/embedchainenv/lib/python3.9/site-packages/embedchain/embedchain.py", line 198, in add
    documents, metadatas, _ids, new_chunks = self._load_and_embed(
  File "/Users/mbp16_2022_m1max/MyTests/python/embedchainenv/lib/python3.9/site-packages/embedchain/embedchain.py", line 363, in _load_and_embed
    embeddings_data = chunker.create_chunks(loader, src, app_id=app_id, config=add_config.chunker)
  File "/Users/mbp16_2022_m1max/MyTests/python/embedchainenv/lib/python3.9/site-packages/embedchain/chunkers/base_chunker.py", line 31, in create_chunks
    data_result = loader.load_data(src)
  File "/Users/mbp16_2022_m1max/MyTests/python/embedchainenv/lib/python3.9/site-packages/embedchain/loaders/web_page.py", line 26, in load_data
    response.raise_for_status()
  File "/Users/mbp16_2022_m1max/MyTests/python/embedchainenv/lib/python3.9/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://twitter.com/elonmusk

Motivation, pitch

Nice elonmusk bot, but make him current: Please also support RAG for social media such as twitter etc

[deshraj]

Hello @yosun, thank you for raising this issue. Your suggestion to include social media platforms like Instagram and X (substitute for Twitter) in our elonmusk bot example is insightful. However, integrating these platforms have limitations on how much data you can fetch but this definitely doable.

We will try to prioritize this soon.