The road ahead

[gorhill]

• [x] Project forked and renamed µMatrix
  • To emphasize companionship with µBlock
  • They complement each other:
    • µBlock for mainstream protection (pattern-based filtering, cosmetic filtering, i.e. ABP-like)
    • µMatrix for advanced protection (i.e. RequestPolicy/NoScript-like) and more advanced privacy features
• [x] Remove all pattern-based and cosmetic filtering
  • [x] Remove associated code
  • [x] Remove associated assets
• [x] Rewrite matrix filtering engine
  • [x] Support global scope rules as finer-grained ubiquitous rules
    • Huge amount of work: easier said than done
  • [x] Convert all dependent code to new scoping mechanism
• [x] Remove settings:
  • [x] "Copy all rules from global scope into newly created local scopes"
  • [x] "Auto create temporary {domain|site}-level scope"
  • [x] "Auto delete unused temporary scopes"
• [x] Remove "Ad blocker-like" from preset setups
• [x] Implement same mechanism as µBlock to fetch remote lists of blocked hosts
• [x] Rewrite popup.js without jQuery (because mem footprint of popup)
  • The more we bring the popup, the more base memory footprint grows. I suspect this will help improve that, if not fix it.
• [x] Create a Crowdin project for µMatrix

Meanwhile, what is best to do IMO:

• Go to HTTPSB's "Ubiquitous rules" in the dashboard:
  • Uncheck "Parse and enforce Adblock+ complex filters"
  • Uncheck "Parse and enforce Adblock+ element hiding filters"
  • Uncheck all lists of ABP-compatible filters:
    • EasyList
    • EasyPrivacy
    • All lists from Spam404 and below
  • Click "Apply changes"
• Install µBlock
  • From now on, let µBlock deals with the pattern-based filtering
  • It's better optimized and has many features not available in HTTPSB:
    • Element hiding helper
    • Collapse blocked elements
    • Can subscribe to external lists
• You will find that HTTPSB runs pretty lean with ABP-filtering disabled
• You may want to disable "Malware domains" lists in µBlock and rather let HTTPSB deals with these.
  • HTTPSB is much more strict when it comes to blocking, i.e. it will block all requests to blacklisted hostnames, while µBlock never block net requests for the root documents (in line with ABP's semantic).
  • This will cause µBlock to run even leaner.

2014-10-28: µMatrix alpha released.

[ghost]

Sounds good - although it's a pity, IMHO, that the well-received name HTTPSB would vanish ...

One question: Why do you want to remove the "Auto create temporary {domain|site}-level scope" setting?

[harshanvn]

i like uMatrix name, and it seems to be a more streamlined version and designed to work inconjunction with ublock, i would definitely use it. And it would reduce the tinkering of which lists to enable/disable when ublock is installed :) The main feature currently i am looking in uMatrix is HTTPSB's privacy feature set, which i miss in uBlock.

[gorhill]

Why do you want to remove the "Auto create temporary {domain|site}-level scope"

Generally: at some point, too many settings is more confusing than not. Many new users end up not having an idea how to best use the extension. I need to enforce a preferred way to use it. The less settings the less worries about how to use the extension optimally.

Specifically: I was using HTTPSB, err... uMatrix with the auto-creation enabled. If uMatrix is going to support global rules as ubiquitous rules, enabling auto-creation will get in the way, it will become annoying, because it will get in the way of creating global rules when landing on a new page. I vist new page, I see linkedin.com in the matrix, I want to blacklist this tracker one globally, but with a scope auto-created, I need to switch back to global, than back again to scope. It's annoying.

The "normal" way of using uMatrix in block-all/allow-ex approach will be to always browse in global scope, and exceptionally users can create a scope for when it's a web site they visit often. Given scopes are exceptions, there shouldn't be a setting to auto create every single time. Out of the box a user will end up creating a lot of scopes, but once his environment is setup in uMatrix, browsing in global scope becomes optimal.

I consider now scope creation to be an exception, and having one created each time a new web site is visited doesn't make much sense. It also clutters the memory, it forced me to put in there yet another option to garbage collect unused scopes.

[ghost]

I consider now scope creation to be an exception

I admit that I don't understand this yet. The beauty of the domain/site-level scopes is that those services like googleapis and the likes which are nearly omnipresent, can be selectively whitelisted where absolutely needed and thus "sandboxed" in order to prevent user-tracking. How can this be achieved if a global scope is the normal way of using µMatrix?

I guess that my lack of understanding this might be related to my not understanding what

Support global scope rules as finer-grained ubiquitous rules

exactly means ...

[gorhill]

I didn't say scopes are going away, I said there just won't be a setting to have them auto-created each time a user land on a page with a new domain for which a scope doesn't exist.

exactly means

The option "Copy all rules from global scope into newly created local scopes." will become unnecessary: all rules in global scope will be seen by narrower scopes without the need to copy them internally.

[ghost]

I didn't say scopes are going away, I said there just won't be a setting to have them auto-created each time a user land on a page with a new domain for which a scope doesn't exist.

I know. I just think that many users prefer a narrower scope by default for privacy reasons. In the future one has to remember to manually switch to a narrower scope if one stumbles over a site that contains those omnipresent 3rd party services. That setting is simply more convenient, IMHO.

[gorhill]

That setting is simply more convenient

• Have auto-creation enabled
• Land on some page
• Scope created
• See linkedin.com in the matrix
• Blacklist linkedin.com
• Oh wait, I want to blacklist this one everywhere
• Switch back to global scope
• Blacklist linkedin.com
• Persist changes
• Scope gone

That's wasn't convenient.

Thing is, I have to look at the big picture, i.e. what is best in my opinion for a majority of users. I just wish we all have this bigger picture in mind when requesting features. I do go once in a while on a google search to find out the unformal feedback about the extension. And one thing that sticks out in my mind is people aren't too sure how to use it optimally. To fix that is to avoid having so many settings that users end up uneasy about whether they are using it properly or not. I am trying to figure how to fix that.

In block-all/allow-ex mode, scopes are exceptions, so the exceptional step is to click to create one. In allow-all/block-ex mode, scopes are irrelevant.

If ever a user want to whitelist a whole page, which is the typical case of having a scope when working in block-all/allow-ex mode, that would be ctrl-A, which will always create a scope in such case if none exists (I could add an icon button for this one since the ABP icon is gone).

[ghost]

Well, I think it depends on how HTTPSB is used. I'm using it in block-all/allow-ex mode which means that linkedin.com is blocked (greylisted) by default. So normally no need to explicitly blacklist that kind of domains (steps 5 and 6 in your list above). And trackers etc. are blacklisted via hosts files anyhow.

I guess it's a matter of taste. I certainly acknowledge your points but for users who do not see scopes as exceptions but deliberately as default behavior, the lack of that setting just makes it a bit more complicate to get what they want. Nevertheless, I think that those users can be okay with these changes.

[gorhill]

So normally no need to explicitly blacklist that kind of domains

Disagreed. There is css/img, and if whitelisting the all cell, happy to have it blacklisted expressly.

I guess it's a matter of taste

I decided to address things this way now: there is a scale: sites broken = less users, sites not broken = more users. So I am addressing things from that perspective. uBlock breaks less than uMatrix allow-all/block-ex which breaks less than uMatrix block-all/allow-ex. To not overcomplicate for the majority of users, in order to not discourage but rather nurture good privacy/security habits.

uMatrix will never become less powerful, I just want to make it more accessible to more. Too many settings which are meaningful to only one end of the scale. Auto-scope creation is useful only to block-all/allow-ex, and only a portion of those users of that mode, and even for those users, it can get in the way sometimes.

[ghost]

I think we could completely do away with some of the options if some of the functionality is handled different.

If all scopes could read global without copying them, that would help a lot.

I am a Block-All/Allow Ex user, and I have these set:

1. Auto create temporary domain level scope.
2. Copy all rules from global scope into newly created local scopes.
3. Auto delete unused temporary scopes.

If local could read global without a copy then #2 option is no longer needed, and it also keeps the individual scopes shorter (for example I allow Youtube Videos but no cookies everywhere, but every new scope has them, and I had to add them to older scopes). I expect to have to open the matrix as a block all user, at the very least to see, and we could just have me setting things there create the scope in #1.

[gorhill]

If local could read global without a copy

Currently working on this. uMatrix won't come out without this. It's a huge amount of code to rewrite, but it's the aspect of HTTPSb which bothers me the most. It will be definitely fixed, I can't tell how long it will take.

[stormy-henderson]

I have extensively customized and then ditched HTTPSB several times. I've decided the only way I can use it and be happy is to block-all/allow-ex and then auto-create & auto-whitelist the page domain for JS/XHR. This lets me browse 95% of pages without errors and only edit the matrix when I need cookies or shopping carts, while still blocking all the third-party junk.

Will losing "Auto create temporary {domain|site}-level scope" mean we lose auto-whitelisting of the page domain?

And while I have often had to do the annoying changing-scopes dance you describe above to allow or block something globally, I far more often want to edit the matrix for the page domain that I am browsing, and having the matrix display the page domain by default makes for the easiest & quickest use, and is more likely to encourage correct usage by new users than default global scope.

I still believe a layered, inheriting matrix is needed for this software to work most powerfully and understandably (see https://github.com/gorhill/httpswitchboard/issues/227).

Thanks for the great extension!

[gorhill]

I far more often want to edit the matrix for the page domain that I am browsing

I currently use the matrix in allow-all/block-ex, except I blacklist cookie, XHR and frame in global scope. Whenever I land on a site which I want to unbreak for good, it is quite trivial to:

1. while in global, first blacklist all that deserve it that was not done yet
2. switch to domain-level scope
3. unblacklist cookie, xhr
4. save, done

As of now it's really the easiest way I have found to surf with good peace of mind, little breakage, and where breakage exists in need of unbreakage, it is trivial. This is my whole point: I figure this way HTTPSB becomes more within reach of people who are less tolerant of site breakage.

I am trying to lower the annoyance point at which less tolerant users give up on a tool beneficial for them. This is just an undeniable reality: hardcore users who block-all/allow-ex by default have a higher tolerance to security/privacy tool annoyance, I am trying to help the ones who have a lower tolerance so they can benefit from such tool too.

[ghost]

I had no intended to prolong this discussion. On the other hand, you're in a crucial decision phase regarding how µMatrix will look like. Thus, I think it makes sense to bring one's arguments forward. Sooo ...

1. while in global, first blacklist all that deserve it that was not done yet
2. switch to domain-level scope
3. unblacklist cookie, xhr
4. save, done

I'm afraid that many users will simply not think about step 2 above. Remember a recent example where a user wasn't aware of the Element Picker in µBlock and thought that the Power Button was to disable µBlock everywhere. I think this kind of inexperienced/new users will predominantly use the global scope in µMatrix and nothing else. And this has nothing to do with the fact if the "Auto create temporary {domain|site}-level scope" setting exists or not. They might have used Noscript before which doesn't have scopes at all (if we ignore rather complicated ABE rules). In any case, it's necessary to teach those users through a streamlined documentation or constructive descriptions in µMatrix about the benefits of using narrower scopes.

Now you're arguing that it wasn't convenient to switch from a narrower scope to the global scope just in order to blacklist domains like linkedin.com. I still question the need to do this very often. Practically all relevant adservers/trackers/malware domains are blacklisted through the ubiquitious lists anyhow. I'm not saying that all other 3rd party domains not contained in those lists are not evil - but possible security/privacy concerns are very limited as long as only css / img are whitelisted. I agree that whitelisting the all cell is a problem - that's why I suggest to add a warning (popup) that with this action all (not explicitly blacklisted) 3rd party domains are whitelisted, too, which might result in privacy/security problems.

So my point is: The need to explicitly blacklist an already blocked/graylisted domain is considerably smaller than the need to whitelist specific 3rd party domains in order to make a website work properly. And a privacy/security-conscious user wants to this in a site/domain-level scope (I think this is what @stormy-henderson said above.) This is one of the things which makes HTTPSB so unique compared to, say, Noscript: Its sophistication, its explicit unequal treatment of websites if it comes to whitelisting 3rd party domains. You said that this funtionality will still exist. Okay - but I'm afraid that even "hardcore" users, as you put it, will find out at the end of the day more frequently than they would like to see that they had accidentally whitelisted some 3rd party domains in the global scope simply because they had forgotten to switch to a narrower scope before. So they would have to delete that rule in the global scope, surf to the relevant website(s), switch to a narrower scope and save the rule(s) therein again. Not very convenient but rather error-prone instead.

To cut a long story short:

1. Newbies/unexperienced users will most probably use the global scope only. They will not benefit from the removal of that setting. On the contrary, its mere existence might even make some of them familiar with the concept of scopes - provided that a documentation with some constructive examples explains their usage. It's probably also useful to subsume this setting under "Advanced settings" which should be clearly separate from other settings.
2. Experienced users who go in for creating scoped rules will seriously miss that setting as its absence makes operating µMatrix less convenient and more error-prone.

Anyways - it's your decision, Raymond. We will accept whatever you decide to do :)

[gorhill]

crucial decision phase regarding how µMatrix will look like

Look, all can be changed afterward according to feedback etc. It's a fork, HTTP Switchboard will still be there. I just want to take steps back and focus on people who could probably use it, but got confused by the plethora of choices right off the bat.

They will not benefit from the removal of that setting

They benefit in the way of them not ending wondering whether they should use it or not. One middle way solution would be to have an advanced button which would unveil more settings which would be more suitable to advanced users.

I want to provide a basic setup which will not scare away new users. That the extension "takes time to figure" is the most common comment I see. That bothers me because I know it's possible to make it a "work out of the box" extension with little site breakage. (Ex: http://www.nsaneforums.com/topic/224243-list-your-favorite-extensions-and-add-ons/page-2#entry816356)

I just want current advanced users to take a more altruistic view and work with me to bring this tool to more people, there is so much to benefit from it. That means forgetting about our own pet habits as advanced users and trying to help those who would like to use it but feel like they need to take a course to use it.

[ghost]

HTTP Switchboard will still be there

Will you still work on it and improve or will it just stay as it is right now?

I want to provide a basic setup which will not scare away new users. That the extension "takes time to figure" is the most common comment I see.

I understand that and I appreciate that you want to make it easier. However, I doubt that removing a useful setting is really helpful. It's better - as already mentioned - to "hide" it in order to make it clearly available only to advanced users.

I just want current advanced users to take a more altruistic view and work with me to bring this tool to more people

I'm trying hard but I'm coming to other conclusions ;-) For example, you said that domain/site-level scopes will still be available. So you assume that those scopes will be actually used at least by some people (otherwise it would be logical to abandon them). Those scopes will be used if the users understand them and their benefits. Once this is the case, they will not be scared away by a "hidden" setting but will be happy to have it available. And those users who will only use the global scope will probably not care about that setting.

As a general remark, I'm convinced that it's not the (number of) settings in the first place why people think that HTTPSB "takes time to configure". I rather think that it's the matrix which is regarded as more complex than, e.g., Noscript. You wrote about the alleged "need for more micromanagement" here. Even Noscript is considered by many users as too complex. And HTTPSB is a combination of Noscript and RequestPolicy and much more. For Noscript it's a simple question: Whitelist or not? In HTTPSB, however, there are six columns which are black-/graylisted - Gosh, how complicated! ;-) . How many users really know that simply clicking the hostname cell solves this "problem"?

So you can either make HTTPSB/µMatrix less powerful - I don't think that's what you want. Or there should be a more guided, streamlined step-by-step documentation in combination with more comprehensive descriptions (and possibly reorganized settings) in the extension itself in order to help the new users to use the extension according to their needs. I'm abolutely willing to help, if you want, and I'm sure that other users will join as well.

[gorhill]

Or there should be a more guided, streamlined step-by-step documentation

Definitely needed. I picture something extremely simple, very short videos, or a series of snapshots of typical way to "solve" particular problem. But to create these consume huge amount of time. Screenshots, etc. The amount of work is discouraging, and even if I could get it all done, many users have their mind made that it's too complicated, and spreading the ideas that it is.

The idea I had this morning to to scrap the setup screen. Out of the box uMatrix will come out in allow-all/block-ex mode, but with at least all lists enabled. Site breakage should be quite on the low side. Than already I am happy that users can at least see how they are being abused (number of hosts hit). From this point, they can start slowly familiarizing themselves with it. So long as it doesn't break everything out of the box is key at this point I think.

For an advanced user, it would be just a matter of clicking the all cell to turn uMatrix back to a block-all/allow-ex mode. I doubt that would bother advanced users.

[hrj]

@gorhill I appreciate your concern for usability, but the micro-decisions don't seem to align with the project's goal of increasing privacy.

You seem to be optimising for the rare case (considering the intended audience of privacy concerned users). From my perspective, site breakage should be a rare case. Use of global scope ought to be a rare case too.

Moreover, you already have a wizard-mode of sorts (can't find it in the settings right now), which can help new users get up and running quick. This could be leveraged further and shown right at install time.

I am sorry if this is repeating the same arguments, but how else can we show our numbers? :)

[gorhill]

how else can we show our numbers?

Well, the people who abandoned early because of perceived complexity aren't going to show up here to be counted.

[hrj]

Well, the people who abandoned early because of perceived complexity aren't going to show up here to be counted.

Good point, but that could fly either way. Those who abandon it for lack of configurability / privacy might not show up too. I hope that those who show up, do count :)

[ghost]

The idea I had this morning to to scrap the setup screen. Out of the box uMatrix will come out in allow-all/block-ex mode, but with at least all lists enabled. Site breakage should be quite on the low side.

I think that's a good idea.

[mikhaelkh]

@gorhill uMatrix & HTTPSB have some other common subscriptions: malware domains, various hosts lists that have only domains. They are better have enabled only in HTTPSB? Also, in uBlock you can add that Fanboy Annoyances includes Fanboy Social (proof) and BitBlock includes most of Fanboy Annoyances (proof).

[gorhill]

in uMatrix you can add that Fanboy Annoyances includes Fanboy Social (proof) and BitBlock includes most of Fanboy Annoyances (proof)

uMatrix will have only hostname-based lists.

[mikhaelkh]

@gorhill, in uBlock, of course. Fixed previous comment)

[gorhill]

I will throw some ideas just for the purpose of keeping track of these for myself, as I tend to forget often when busy with other stuff.

So re. "auto creation of scope":

• No more need to create a scope, all scopes exist all the time (virtually).
• From the point of view of a user, a user doesn't need to pick which scope is going to be active: the site-level scope always exist, the domain-level scope always exist.
• These domain- and site-level scopes just happen to have the same exact rules as the global scope, unless they are tampered with.
• A copy-on-write approach will cause a real scope to be instantiated in memory whenever its rules differs from the scope underneath.
• A real scope which ends up as having exactly the same rules as the one layered under will be removed from memory.
• So the above gets rid of settings for auto-creation and auto-deletion of scopes.
• UI-wise, need to come up with some visual for the scope selector which makes it clear that the three scopes always exist (and to be able to glance if there are unpersisted temporary rules in any of them).
• Other various ideas re implementation:
  • Instead of a scope being implemented as three separated set of rules (black, white, gray), it will now be just one set of rules, but each rule can have one of four states: black (-1), gray explicit (0), white (+1), or gray implicit (undefined). (make layering algorithm way easier)
  • No need to have data structure for the permanent scopes in memory: a simple string which represents the streamed version of a scope can simply be attached to the instantiated scopes.

[ghost]

• These domain- and site-level scopes just happen to have the same exact rules as global scopes, unless they are tampered with.

Raymond, this sounds very interesting, indeed. However, I don't understand yet how this will look like in practice. How can the user determine under this approach if the rules are meant to be global or site-/domain-specific? Say, I tamper with a site-specific scope - does this mean that the respective global and domain-specific scopes will be deleted? Or will the site-specific scope just take precedence and become the real scope?

[gorhill]

I'm committing 100% to the idea of layering as was expected in issue #227. Other users also expected layering when using HTTPSB.

It's really what you see is what you get. If you explicitly blacklist a cell in global scope, the rule will be in global scope. Scopes above will see that rule, unless they have their own explicit rules for that cell.

Layering already exists actually, this is how ubiquitous rules are implemented. So whatever issues you may see with what is proposed above, is probably present if you consider existing ubiquitous rules.

In layering, for each cell, the three scopes are visited to determine the state of the cell. Than if after all this the cell is evaluated as gray, the usual matrix-based inheritance algorithm is applied.

Say, I tamper with a site-specific scope - does this mean that the respective global and domain-specific scopes will be deleted

No, the cell which you tampered with will override whatever is in broader scopes for that same cell.

[ghost]

Thanks - I'm beginning to understand :-)

[gorhill]

Also by the way, given the way I now plan to implement scopes etc as described above, I have furtehr thought:

The auto-whitelist feature will always apply to the narrowest scope, just like it is currently the case if you whitelist through [Alt-A].

Also, as said, the three scopes always exist at all time, so the scope which will appear when you bring up the matrix will just be the last one which was active last time you accessed the matrix. So if a user tampers mainly with site-level scopes, the automatically selected scope will be site-level.

[ghost]

Also, as said, the three scopes always exist at all time, so the scope which will appear when you bring up the matrix will just be the last one which was active last time you accessed the matrix.

Hey - that's great!