Used a regex expression for simple URL validation and added tests

[OmriGM]

Background

• Used a regex expression for simple URL validation
• Added additional tests to url validators

Changes

The expression before:

url.startswith("http://") and not url.startswith("https://"):

The expression after:

re.match(r"^https?://", url):

Also, added unit test to tests/unit/test_url_validation.py module, specifically to validate_url() which is the main function in the module

Documentation

Test Plan

PR Quality Checklist

• [x] My pull request is atomic and focuses on a single change.
• [x] I have thoroughly tested my changes with multiple different prompts.
• [x] I have considered potential risks and mitigations for my changes.
• [x] I have documented my changes clearly and comprehensively.
• [x] I have not snuck in any "extra" small tweaks changes

[vercel[bot]]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	May 15, 2023 10:54pm

[codecov[bot]]

Codecov Report

Patch coverage: 100.00% and project coverage change: +0.01 :tada:

Comparison is base (bf33f4a) 62.66% compared to head (469d6ef) 62.67%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #3763      +/-   ##
==========================================
+ Coverage   62.66%   62.67%   +0.01%     
==========================================
  Files          74       74              
  Lines        3426     3427       +1     
  Branches      504      504              
==========================================
+ Hits         2147     2148       +1     
  Misses       1123     1123              
  Partials      156      156              

Impacted Files	Coverage Δ
autogpt/url_utils/validators.py	93.10% <100.00%> (+0.24%)	:arrow_up:

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

[Boostrix]

this sort of regex-based validation routine would also be useful in other places, like file name/path validation (for shell command execution) A number of folks keep reporting that Auto-GPT is trying use absolute paths and even the ~ (home dir) inside the workspace.

[OmriGM]

@ntindle BTW, there is a failing test (tests/integration/test_execute_code.py: test_execute_python_file()) that is dependent on the docker daemon running on the local machine, was that on purpose? Here is the error message of the failing test (before I started the docker daemon on my local computer)

./tests/integration/test_execute_code.py::test_execute_python_file Failed: [undefined]assert 'Error: Error...directory\'")' == 'Hello udaxihhexd!\n'
  - Hello udaxihhexd!
  + Error: Error while fetching server API version: 500 Server Error for http+docker://localhost/version: Internal Server Error ("b'dial unix docker.raw.sock: connect: no such file or directory'")
python_test_file = '/private/var/folders/58/jb5nf1y51b1btsjnnsn9jsdc0000gn/T/pytest-of-omrigrossman/pytest-13/test_execute_python_file0/home/users/monty/auto_gpt_workspace/tmph3oy_5_j.py'
random_string = 'udaxihhexd'

    def test_execute_python_file(python_test_file: str, random_string: str):
        result = sut.execute_python_file(python_test_file)
>       assert result == f"Hello {random_string}!\n"
E       assert 'Error: Error...directory\'")' == 'Hello udaxihhexd!\n'
E         - Hello udaxihhexd!
E         + Error: Error while fetching server API version: 500 Server Error for http+docker://localhost/version: Internal Server Error ("b'dial unix docker.raw.sock: connect: no such file or directory'")

[p-i-]

This is a mass message from the AutoGPT core team. Our apologies for the ongoing delay in processing PRs. This is because we are re-architecting the AutoGPT core!

For more details (and for infor on joining our Discord), please refer to: https://github.com/Significant-Gravitas/Auto-GPT/wiki/Architecting

[anonhostpi]

I think it maybe smarter to use a library for validation. The repository may find URL/URI/Path validation useful in other areas.

[ntindle]

Are we sure Regex is the way to go here? Aren’t there entire libraries for validation?

[Boostrix]

This would at least seem like a start, maybe the OP would like to adopt the feature and extend it over time?

[lc0rp]

@OmriGM would you consider something like this?

urllib:

from urllib.parse import urlparse
parsed_url = urlparse("https://www.google.com")
assert bool(parsed_url.scheme and parsed_url.netloc)

Or the validators package pip install validators

...
import validators
assert validators.url("http://google.com")
...

We'd like to get this into 0.3.2.

Please also update the branch.

[vercel[bot]]

Deployment failed with the following error:

Resource is limited - try again in 16 minutes (more than 100, code: "api-deployments-free-per-day").