mem0 icon indicating copy to clipboard operation
mem0 copied to clipboard

Adding fetching data functionality for reference links in the web page

Open vatsalrathod16 opened this issue 1 year ago • 3 comments

Description

Enhancement: extracts data from the given web page and all the reference link present in that page

Fixes #1234

Type of change

  • [ ] Bug fix (non-breaking change which fixes an issue)
  • [X] New feature (non-breaking change which adds functionality)
  • [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • [ ] Refactor (does not change functionality, e.g. code style improvements, linting)
  • [x] Documentation update

How Has This Been Tested?

Added Unit test and ran for the below script using google embedder and google LLM

import os

from embedchain import App

app = App.from_config(config_path="config.yaml")
app_link = "https://en.wikipedia.org/wiki/Kobbie_Mainoo"
app.add(app_link, all_references=True)

while True:
    question = input("Enter question: ")
    if question in ["q", "exit", "quit"]:
        break
    answer = app.query(question)
    print(answer)

Config

llm:
  provider: google
  config:
    model: gemini-pro
    max_tokens: 1000
    temperature: 0.5
    top_p: 1
    stream: false

embedder:
  provider: google
  config:
    model: 'models/embedding-001'
    task_type: "retrieval_document"
    title: "Embeddings for Embedchain"
  • [X] Unit Test
  • [X] Test Script (please provide)

Checklist:

  • [X] My code follows the style guidelines of this project
  • [X] I have performed a self-review of my own code
  • [ ] I have commented my code, particularly in hard-to-understand areas
  • [ ] I have made corresponding changes to the documentation
  • [X] My changes generate no new warnings
  • [X] I have added tests that prove my fix is effective or that my feature works
  • [X] New and existing unit tests pass locally with my changes
  • [ ] Any dependent changes have been merged and published in downstream modules
  • [X] I have checked my code and corrected any misspellings

Maintainer Checklist

  • [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
  • [ ] Made sure Checks passed

vatsalrathod16 avatar Sep 04 '24 05:09 vatsalrathod16

CLA assistant check
All committers have signed the CLA.

CLAassistant avatar Sep 04 '24 05:09 CLAassistant

Hey @vatsalrathod16 Can you please resolve the conflicts?

Dev-Khant avatar Sep 09 '24 06:09 Dev-Khant

Hey changes looks good but can you please also update the docs for it.

updated the docs.

vatsalrathod16 avatar Oct 01 '24 16:10 vatsalrathod16

Thanks @vatsalrathod16 for the contribution.

Dev-Khant avatar Oct 15 '24 11:10 Dev-Khant