Re-enable ToS;DR Script and update latest data

[JustinBack]

Reviewer:

Description:

I have fixed the tosdr.js script to reflect the current data set uri and skip if it failed to get privacy data. This PR also includes the latest dataset from tosdr.org where as the current state in master is from 2018. I have also re-enabled the execution of the script in the gruntfile to get latest datasets on build.

This PR is more comprehensive than #380 even though the PR by @michielbdejong has essentially fixed the issue.

Closes #482 Closes https://github.com/duckduckgo/Android/issues/973

Steps to test this PR:

Simply compile the extension via make, execute the gruntfile or run the script manually. -->

Automated tests:

• [ ] Unit tests
• [ ] Integration tests

Reviewer Checklist:

• [ ] Ensure the PR solves the problem
• [ ] Review every line of code
• [ ] Ensure the PR does no harm by testing the changes thoroughly
• [ ] Get help if you're uncomfortable with any of the above!
• [ ] Determine if there are any quick wins that improve the implementation

PR Author Checklist:

• [ ] Get advice or leverage existing code
• [ ] Agree on technical approach with reviewer (if the changes are nuanced)
• [ ] Ensure that there is a testing strategy (and documented non-automated tests)
• [ ] Ensure there is a documented monitoring strategy (if necessary)
• [ ] Consider systems implications

[michielbdejong]

@JustinBack thanks for picking this up!

Hm, the build is failing on this PR but probably not due to anything inside it, many other branches are also failng, see https://travis-ci.org/github/duckduckgo/duckduckgo-privacy-extension/branches

[Super-Cow-Pig173]

Good catch

[ruurtjan]

Hey @jonathanKingston @sammacbeth, is anything blocking this merge? Would be nice to have TOS;DR working again :)

[JustinBack]

I have updated the scripts again for our latest API standards. The only disadvantage now is that you'd need an API Key generated by us to get all service data as you hit the hourly rate limit pretty quickly with over 2000 services.

(We need to pull each service one by one to get points data)

This API Key can be entered using the environment variable TOSDR_APIKEY

Currently a lot of case data is saved by caching requests to the case/v1 endpoint, sadly this cannot be done at the service endpoint.

I am open for any suggestions involving changes regarding the ratelimit or script

[JustinBack]

Also not entirely sure if its wise to hardcode Grades into unit tests?

https://github.com/duckduckgo/duckduckgo-privacy-extension/blob/345b525a649d90fcfdd37fd0ed45d7332668c57f/integration-test/background/grades.js#L4-L12

Especially since Google, Reddit, twitter, facebook have a different rating (All E) on ToS;DR. independent is not rated at all, so all PR checks fail

[JustinBack]

The way I currently handle it is by having a "warning" system if the requests are not enough to make a complete file or abort if no file can be created.

[sammacbeth]

@ruurtjan We are currently evaluating this feature internally, which is why we haven't updated it recently. We'll evaluate this PR once we've made a decision.

[saturnonearth]

@ruurtjan We are currently evaluating this feature internally, which is why we haven't updated it recently. We'll evaluate this PR once we've made a decision.

Any update on this?

[ashley0143]

still no updates

[jonathanKingston]

No real change here, it's on our timeline to update; other higher priority work has taken precedence but we're expecting an update in the next few months here.

[JustinBack]

No real change here, it's on our timeline to update; other higher priority work has taken precedence but we're expecting an update in the next few months here.

So, with the removal of ToS;DR: Is this PR stale?