bug: `docs_site` loader does not work

[cachho]

🐛 Describe the bug

Test code:

from embedchain import App
from embedchain.config import ChromaDbConfig, AppConfig

app = App(AppConfig(log_level="DEBUG"),chromadb_config=ChromaDbConfig(chroma_settings={"allow_reset": True}))
app.db.reset()
app.add("https://docs.zextras.com/carbonio-ce/html/general.html#what-is-product")

[Dev-Khant]

@deshraj @cachho Here when I checked for our docs it worked because quickstart was present in all_urls that are fetched in _get_child_links_recursive() of docs_site_loader.py.

But for the given URL in the above code. This current path == carbonio-ce/html/general.html in not present in all_urls and that's why no URL is selected and no data is loaded.

So here to make sure that the given path is present in all_url can we append the current_path to the list? The link response is checked before, so there won't be any issue of a given link not working.

[Dev-Khant]

Hi @deshraj @cachho any update on this?