Focus iOS blobs should likely say "no such feature"

[arthuredelstein]

Pointed out by https://twitter.com/bruudaa/status/1556776696895967232

[Thorin-Oakenpants]

isn't this what I alluded to in #91

when a test breaks a 3p API, then it shouldn't report passed, but blocked

e.g. issue 30

• FF Web SQL Database : both 1p+ 3p results are identical, but they are errors (and IDK it gets a green check mark)
  • so I would mark this as – (no such feature)
• Brave Web SQL .... 1p = results, 3p = Errors = so it's not partitioning the 3rd party, it's blocking it
  • so I would mark this as 🟩 (or some symbol as blocked)
• FF/Brave Cookies: 1p + 3p are valid results
  • if different then passed else failed

Here's a good example: indexedDB

• Brave: 1p works, 3rd party = BLOCKED (not "passed" with a green check mark IMO, although I get that the privacy is achieved - hence a green symbol like 🟩 is appropriate)
• FF: 1p works, 3rd party works = they are different = PASSED
• in this example it looks like Brave + FF are equal: they are in privacy, but not in STATE PARTITIONING TESTS

This should be pretty simply to add a blocked in your pass/fail/no-such-feature logic/function, and then we can actually see what the real differences are

/end of mini rant :)

[arthuredelstein]

Hi @Thorin-Oakenpants -- I didn't fully follow your comments in #91 before, but now I see what you're saying. I'm thinking this problem over some more.

The website currently presents the test results on two levels:

• Level 1 displays three possible values: PASS (), FAIL (), UNSUPPORTED ().
• Level 2 (tooltips) display details about how each test got its result

Level 1 is aimed at end-users (non-technical folks) who just want the bottom line. I want to keep things as simple as possible at this level. End users don't (and shouldn't have to) care whether a feature is partitioned, blocked, unsupported or otherwise mutilated. Rather all that matters is whether or not data leaks between sites. From that perspective, I am tempted to get rid of the UNSUPPORTED option and just show a PASS for those items. (You can see my previous ambivalence in that UNSUPPORTED dashes are already green.)

Level 2, aimed at people knowledgable about Web APIs, is where we should give details about exactly how a test passes or fails, whether the feature is partitioned, blocked, or unsupported. It could definitely do with a lot of cleanup.

You are also right in pointing out that the category names is sloppy. I am thinking about renaming "State Partitioning Tests" to "Cross-site Leak Tests" and editing the description to talk about different protection options including partitioning and blocking.

[Thorin-Oakenpants]

I'm thinking this problem over some more.

Level 2 already shows the diffs, so this is about Level 1 - the instantly seen visuals, keep it simple for users, I understand that that (I still don't really see 4 values as difficult vs 3 values) but I get what you're saying :) Now that I've let your comments percolate in my head for a few days, I think I see an easy way out, and no need to change the subheader, and the change is intuitive

The subheader is STATE PARTITIONING TESTS, if a third party is not supported, then no partitioning is done, and you return UNSUPPORTED (which is green and means you are still protected - i.e not a red X or anything different like a BLOCK which sounds scary)

Thus we can keep just the three possible values, but it will still show the diffs/approaches between browsers, and, technically, is more correct. And now we can see what is actually partitioned vs not - and this will also make it easier to see when something changes or a test fails IMO.

and editing the description to talk about different protection options including partitioning and blocking

you can still do that. A unsupported means cross-site (edit: or first party) is not supported, therefore no privacy issue. Simple huh? e.g.

In these tests ...— means the first party or cross-site party cannot use that method (private), ✓ means the cross-site party content was partitioned (private), and ✕ means the cross-site party content was not partitioned (can leak).

Does that make sense? We're not testing first party, so no one cares if e.g. we have — for Brave's WEBSQL - i.e we're not testing if APIs are supported. This way the three values reflect what is being tested more accurately

[Thorin-Oakenpants]

Rather all that matters is whether or not data leaks between sites. From that perspective, I am tempted to get rid of the UNSUPPORTED option and just show a PASS for those items. (You can see my previous ambivalence in that UNSUPPORTED dashes are already green.)

It's your baby, and if it simplifies your code and makes life easier, then so be it. Personally, I think the level2 is too hidden, but with my suggestion above, keeping three, then it wouldn't be - of course that means you need to fix/update each test to properly detect errors as unsupported rather than passes (you already trap errors)