Antonio De Marinis

icon indicating a website link https://www.linkedin.com/in/antoniodemarinis

icon company @eea icon location Kongens Nytorv 6, 1050, Copenhagen K, Denmark icon location Web Tech Lead at EEA

Results 4 comments of Antonio De Marinis

Docs: How far we want to take it?

I agree. We should not describe in details on how to deploy with specific clustering tools like rancher, swarm, dc/os, kubernetes. We should just explain the basic docker commands. For...

Respect robots.txt / robots meta tag when crawling websites

beside the metatag, the web crawler should also check the robots.txt in the root of the site There seems to be no check about robots.txt in the web connector https://github.com/danswer-ai/danswer/blob/main/backend/danswer/connectors/web/connector.py#L52...

Respect robots.txt / robots meta tag when crawling websites

@Weves Robots.txt is now respected via PR #1538

Secrets: write-up best practices, do's and don'ts, roadmap

I guess what is being done here is kept _secret_ :D