suss-mcx comments

Repositories
Issues
Comments

Results 2 comments of


                                            suss-mcx

[Question]: Why are addresses which resolve to internal IPs considered invalid URLs?

@KevinHuSh In my case the IP the hostname resolved to is `172.22.150.251`, and the crawler failes with "Invalid URL". The mentioned `is_valid_url` function explicitly calls another function from the same...

[Question]: Why are addresses which resolve to internal IPs considered invalid URLs?

@KevinHuSh Well yes, it does, I'm using a hostname WITH a scheme (like `https://`), the fact that it resolves to a private IP is what breaks crawling – as I...