webperf_core icon indicating copy to clipboard operation
webperf_core copied to clipboard
verified publisher icon Webperf-se

Remove Internet access requirement to run local Tests

Open 7h3Rabbit opened this issue 1 year ago • 1 comments

Feature/improvement

Today we require Internet access even for otherwise local tests. This is to get latest information for that specific test. For example CSS test requires access to: https://developer.mozilla.org/en-US/docs/Web/CSS/Reference

We should identify all places that uses get_http_content for this reason and convert it to an github action and store the content in our repo.

This way it will not require Internet access but also make sure we use same references between instances.

7h3Rabbit avatar Aug 13 '24 19:08 7h3Rabbit

Please note: We need to check license requirements for this.

7h3Rabbit avatar Aug 13 '24 19:08 7h3Rabbit

We should add a --credits option in webperf-core that lists all:

  • information sources
  • all projects we depend on
  • all developers who has contributed to webperf-core directly

cockroacher avatar Oct 18 '24 11:10 cockroacher

External URL:s found in webperf_core source:

  • https://api.github.com/repos/{0}/{1}
  • https://api.github.com/repos/{0}/{1}/{2}/{3}/labels
  • https://api.github.com/repos/{0}/{1}/{2}?state=closed&per_page=100
  • https://api.websitecarbon.com/site?url={url}
  • https://cdn.datatables.net/releases.html
  • https://developer.mozilla.org/en-US/docs/Web/CSS/Reference
  • https://developer.mozilla.org/en-US/docs/Web/HTML/Element#obsolete_and_deprecated_elements
  • https://httpd.apache.org/security/vulnerabilities_24.html
  • https://learn.microsoft.com/en-us/lifecycle/products/internet-information-services-iis
  • https://mta-sts.{hostname}/.well-known/mta-sts.txt
  • https://nginx.org/en/security_advisories.html
  • https://nuget.optimizely.com/package/?id=EPiServer.Find
  • https://openssl-library.org/news/vulnerabilities/
  • https://openssl-library.org/policies/releasestrat/index.html
  • https://spdx.org/licenses/
  • https://svn.apache.org/viewvc/httpd/httpd/tags/
  • https://wordpress.org/plugins/{0}/advanced/
  • https://www.cvedetails.com/vulnerability-list.php?vendor_id=26&product_id=3427&page=1
  • https://www.php.net/eol.php
  • https://www.php.net/releases/

cockroacher avatar Oct 30 '24 14:10 cockroacher

Hi @7h3Rabbit and @marcusosterberg, I hope you’re both doing well! I wanted to check in and see if you have any feedback regarding the current status of "credits". Your insights would be greatly appreciated as I move forward. See below:

Credits

Following shows projects and contributors for webperf-core and its dependencies. Many thanks to all of you! :D

webperf-core

License: MIT Contributors:

  • 7h3Rabbit
  • cockroacher
  • flowertwig-org
  • dependabot[bot]
  • marcusosterberg
  • krompaco
  • puggan
  • linulas

lighthouse

License: Apache-2.0

pa11y

License: LGPL-3.0-only usage: Used in Accessibility (Pa11y) Test

sitespeed.io

License: MIT usage: Used in the background in most cases where we need to visit website as browser

vnu-jar

License: MIT usage: Used in HTML and CSS Validation Test

yellowlabtools

License: GPL-2.0 usage: Used in Quality on frontend (Yellow Lab Tools) Test

External Information Source(s):

CSS Validation Test Sources:

  • https://developer.mozilla.org/en-US/docs/Web/CSS/Reference

HTML Validation Test Sources:

  • https://developer.mozilla.org/en-US/docs/Web/HTML/Element#obsolete_and_deprecated_elements

Software Test Sources:

  • https://api.github.com/repos/{owner}/{repo}
  • https://api.github.com/repos/{owner}/{repo}/{source}/{versions_dict[version][id]}/labels
  • https://api.github.com/repos/{owner}/{repo}/{source}?state=closed&per_page=100
  • https://cdn.datatables.net/releases.html
  • https://httpd.apache.org/security/vulnerabilities_24.html
  • https://learn.microsoft.com/en-us/lifecycle/products/internet-information-services-iis
  • https://nginx.org/en/security_advisories.html
  • https://nuget.optimizely.com/package/?id=EPiServer.Find
  • https://openssl-library.org/news/vulnerabilities/
  • https://openssl-library.org/policies/releasestrat/index.html
  • https://spdx.org/licenses/
  • https://svn.apache.org/viewvc/httpd/httpd/tags/
  • https://wordpress.org/plugins/{name}/advanced/
  • https://www.cvedetails.com/vulnerability-list.php?vendor_id=26&product_id=3427&page=1
  • https://www.php.net/eol.php
  • https://www.php.net/releases/

cockroacher avatar Oct 30 '24 20:10 cockroacher

@cockroacher Cool! Looks good :)

Some feedback:

  • projects we depend on should probably have a URL on them
  • Nice grouping urls used by test
  • We should probably have write more for every url/domain? License, maintained by or something?
  • we should list packages from requirements.txt (python dependencies) also
  • I can't find any URL:s to digg.se (should be some from the A11y Statement test?)
  • IP2Location package and file should be here (For GDPR related info)
  • We should have https://github.com/github/advisory-database here, as we use it for software test (but through part of repo and not url)
  • We should have https://blocklistproject.github.io/Lists/alt-version/tracking-nl.txt used in Tracking Test
  • We should have https://blocklistproject.github.io/Lists/alt-version/ads-nl.txt used in Tracking Test
  • We should have https://raw.githubusercontent.com/disconnectme/disconnect-tracking-protection/master/services.json used in Tracking Test
  • We should have https://webbkoll.5july.net/ as we use it for WebbKollen test.

7h3Rabbit avatar Oct 30 '24 20:10 7h3Rabbit

@7h3Rabbit @marcusosterberg

Following has been fixed:

  • projects we depend on should probably have a URL on them
  • we should list packages from requirements.txt (python dependencies) also

Following is on todo:

  • We should probably have write more for every url/domain? License, maintained by or something?
  • We should have https://github.com/github/advisory-database here, as we use it for software test (but through part of repo and not url)
  • We should have https://blocklistproject.github.io/Lists/alt-version/tracking-nl.txt used in Tracking Test
  • We should have https://blocklistproject.github.io/Lists/alt-version/ads-nl.txt used in Tracking Test
  • We should have https://raw.githubusercontent.com/disconnectme/disconnect-tracking-protection/master/services.json used in Tracking Test
  • We should have https://webbkoll.5july.net/ as we use it for WebbKollen test.

cockroacher avatar Nov 01 '24 18:11 cockroacher

Following has been fixed:

  • We should have https://github.com/github/advisory-database here, as we use it for software test (but through part of repo and not url)
  • We should have https://blocklistproject.github.io/Lists/alt-version/tracking-nl.txt used in Tracking Test
  • We should have https://blocklistproject.github.io/Lists/alt-version/ads-nl.txt used in Tracking Test
  • We should have https://raw.githubusercontent.com/disconnectme/disconnect-tracking-protection/master/services.json used in Tracking Test

cockroacher avatar Nov 01 '24 18:11 cockroacher

Credits

Following shows projects and contributors for webperf-core and its dependencies. Many thanks to all of you! :D

Contributors:

Projects we depend on:

lighthouse

Usage: Used in Google Lighthouse based Tests License: APACHE-2.0

pa11y

Usage: Used in Accessibility (Pa11y) Test License: LGPL-3.0

sitespeed.io

Usage: Used in the background in most cases where we need to visit website as browser License: MIT

vnu-jar

Usage: Used in HTML and CSS Validation Test License: MIT

yellowlabtools

Usage: Used in Quality on frontend (Yellow Lab Tools) Test License: GPL-2.0

beautifulsoup4

Usage: Used to parse HTML content

lxml

Usage: Used to parse XML content

requests

Usage: Used to request content not normally requested by users, like robots.txt

urllib3

Usage: Used to parse url

dnspython

Usage: Used to get and parse DNS content

certifi

selenium

IP2Location

Usage: Used to get a proximate location related to IP/IP-range

Pillow

Reguired by sitespeed.io

OpenCV-Python

Reguired by sitespeed.io

Numpy

Reguired by sitespeed.io

cryptography

pylint

Usage: Used by our devs to ensure code quality

packaging

Usage: Used to understand and compare software versions

External Information Source(s):

Update HTML and CSS Test Sources:

  • https://developer.mozilla.org/en-US/docs/Web/CSS/Reference
  • https://developer.mozilla.org/en-US/docs/Web/HTML/Element#obsolete_and_deprecated_elements

Update Software Test Sources:

  • https://github.com/github/advisory-database/tree/main/advisories/github-reviewed
  • https://api.github.com/repos/{owner}/{repo}
  • https://api.github.com/repos/{owner}/{repo}/contributors
  • https://api.github.com/repos/{owner}/{repo}/{source}/{versions_dict[version][id]}/labels
  • https://api.github.com/repos/{owner}/{repo}/{source}?state=closed&per_page=100
  • https://cdn.datatables.net/releases.html
  • https://httpd.apache.org/security/vulnerabilities_24.html
  • https://learn.microsoft.com/en-us/lifecycle/products/internet-information-services-iis
  • https://nginx.org/en/security_advisories.html
  • https://nuget.optimizely.com/package/?id=EPiServer.Find
  • https://openssl-library.org/news/vulnerabilities/
  • https://openssl-library.org/policies/releasestrat/index.html
  • https://spdx.org/licenses/
  • https://svn.apache.org/viewvc/httpd/httpd/tags/
  • https://wordpress.org/plugins/{name}/advanced/
  • https://www.cvedetails.com/vulnerability-list.php?vendor_id=26&product_id=3427&page=1
  • https://www.php.net/eol.php
  • https://www.php.net/releases/

Tracking and Privacy Test Sources:

  • https://blocklistproject.github.io/Lists/alt-version/tracking-nl.txt
  • https://blocklistproject.github.io/Lists/alt-version/ads-nl.txt
  • https://raw.githubusercontent.com/disconnectme/disconnect-tracking-protection/master/services.json

cockroacher avatar Nov 02 '24 22:11 cockroacher