Switch UA for every request (within a well-defined set)

[madduck]

trafficstars

Would it be possible to add a new mode to this nice extension, namely to randomly assign a new user-agent for every request, incl. sub-requests, choosing among a well-defined set of user-agents?

It would be even better if it added some form of fuzzing, i.e. taking a string like

Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0

and modifying the numbers a bit, i.e. turning it into "Firefox/60.1" or "Gecko/201421112", i.e. only choose higher numbers?

[madduck]

https://github.com/LeoTindall/randomua does a good job at this!

[ray-lothian]

FAQ9? https://add0n.com/useragent-switcher.html

[madduck]

No, that's not what I am after, and I am sorry that I wasn't clear.

Like RandomUA, I'd like to get my browser to send a random user-agent on every single HTTP request, whether that is to the same domain, or a different one.

To do so, I'd define a pool of user-agent strings, and then for every HTML page, ever CSS resource, every image, and every script, a random entry from that pool is selected, ideally fuzzed (like RandomUA), and sent along.

You might ask "why not just use RandomUA", and that is a legitimate question.The main reason is that RandomUA seems to operate on a hard-coded set of strings, while I really appreciate the UI and the breadth of your add-on. I did file https://github.com/LeoTindall/randomua/issues/7 about this, but I think this functionality is good enough that users of your extension might also benefit from this.

[ray-lothian]

If I recall correctly this works;

{
  '*': ['ua_1', 'ua_2', 'ua_3']
}

[madduck]

Right, I tried that, and I still think that the actual username chosen gets pinned. Put differently, I just ran a short experiment with 2 user-agents defined in JSON and issues 1,000 requests to a handful of servers, all of them had the second user-agent in the request headers, and the chance of that happening is quite low ;)

Anyway, I suspect that there may be a bug in the random selection of the entry in the list to pick because they were all the same!? I'll take a look…

[madduck]

So turns out that loading a page with sub-requests (I just used Github…) would only actually execute the common.js:match() function once. No wonder then that the user-agent is the same for all sub-requests.

Researching this a bit further, it seems that this is by design. The logic starting at common.js:191 seems to cache the user-agent for each tab.

And in fact, if I replace the entire caching logic and have match() be called for ever request, then I get the desired result, at least it seems to work.

So the questions are:

1. why are we caching the user-agent for a tab?
2. is the use-case wherein we do not cache, but call match() on every request even one that you are willing to entertain for this extension?

[ray-lothian]

why are we caching the user-agent for a tab?

1. WebRequest is a sync operation. We need to have a very efficient function here. That caching thing helps reducing the computational cost of the sync operation.
2. It normally makes no sense to change your UA string for a single tab. This way websites can easily distinguish that you have an extension that is altering your UA string.

In the latest commit, I added a new option in the options page for what you need. Please give it a try and report back

[madduck]

Hey @ray-lothian, thanks for taking your time to engage with me over this, and I can right out confirm that 166e756a45326f110bf85e6655449ed40d4d7bbc does indeed make the intended behaviour configurable.

I do wonder if there is really a computational hit. Assuming cache access uses a hash, that should have a complexity of somewhere between O(1) and O(log n) in Javascript. Array indexing, however, is O(1), always, and essentially picking a random user-agent is nothing other, or is it? Also, random number generation should be quick and constant, and if that's not the case, we could simply use nstimestamp % array length, which would be sufficiently random for the purpose. So even if we had 1,000 user-agent strings, picking a random one should be at least as fast as using a cache, or am I missing something here?

Regarding you responses:

How would you feel if I hacked away at a fourth mode, i.e. blacklist/whitelist/custom, and "random", which would really just take the list of user-agents the user selects using the UI dialog, and make it available randomly?

I do believe there is a good use-case for this, but obviously everyone's mileage varies.

I look at it this way: A tab can be reused, and it's thus not "a bounded unit of browsing activity". Also,0 every web page one loads today issues countless subrequests to other domains, including trackers, and having a constant user-agent string throughout all of this gives trackers an additional data point they can use for correlation.

On the flip-side, you say that webseits can easily identity when an extension is altering the UA string. While this is obviously true in the presence of e.g. an individualised cookie, I for one doubt that many go through the trouble to analyse this, nor will it work across domains.

Anyway, I really appreciate your extension, and your willingness to entertain my thoughts and blather. I'll see what I can come up with, and will submit a PR, ok?

[madduck]

Hm, still trying to understand your codebase. Is there an easy way to get a list of all user-agents, or how would you extend the ua object with a method like random()? I mean, I can write that method of course, but I cannot figure out where all the available user-agents are stored.

[madduck]

I think I am beginning to understand that your approach is quite different, and maybe it won't be quite as easy to do what I had in mind.

Correct me if I a wrong, but asI see things now, the UI is used to select one UA to be assigned to the window (i.e. global), or to tthe current and future tabs (i.e. "apply"). Black/whitelist determines when the UA is even overridden. Custom mode is separate.

If I wanted to implement "random" mode, i.e. select a new UA from the list in data/popup/list.json for every tab/domain/request, then I'd somehow need to import that list into memory, which is quite a bit too much, I'd say.

So now we have to pick a subset of user-agents, don't we? In which case my use case if probably fringe enough to warrant just using custom mode for that.

I think your commit goes a long way. Can we talk naming a bit though? Instead of "use caching to improve performance" how about "Keep the same user-agent string as long as a tab is loaded. If unchecked, this means that a new user-agent string is chosen at random for every request."

And then, "random mode" should really become a fourth mode, with its own configuration textarea to allow me to specify a list of user-agent strings without having to use JSON, right?

[ray-lothian]

I do wonder if there is really a computational hit. Assuming cache access uses a hash, that should have a complexity of somewhere between O(1) and O(log n) in Javascript

For every single URL, we do have this hostname parsing and that part is an additional step which I am trying to bypass. To improve the speed, I even had to use a custom function to get the actual hostname. The new URL is really slow for this purpose. For the custom mode, we need to have it anyway. But in your case definitely not needed. So I would say let's have a new mode and get rid of that caching preference. In the UI we can have a new button to add a UA string to the random list.

How would you feel if I hacked away at a fourth mode, i.e. blacklist/whitelist/custom, and "random",

would be a perfect solution.

I for one doubt that many go through the trouble to analyse this

Even if you don't want to keep the UA string fixed for the tab, still it makes more sense to have a single UA string per frame, not per request. At least we can have this as an option for the random mode.

will submit a PR, ok?

Sure!

If I wanted to implement "random" mode, i.e. select a new UA from the list in data/popup/list.json for every tab/domain/request, then I'd somehow need to import that list into memory, which is quite a bit too much, I'd say.

We need to have it in the memory. This is the fastest possible solution, but we do not need to have the list.json. It is user responsibility to include all the needed UA string in the options page. I would say define a new preference. A user can add into this list from the options page or from the toolbar popup. In the background, the list automatically gets updated by https://github.com/ray-lothian/UserAgent-Switcher/blob/master/common.js#L31.

And then, "random mode" should really become a fourth mode, with its own configuration textarea to allow me to specify a list of user-agent strings without having to use JSON, right?

Looks good to me.

[madduck]

Hey @ray-lothian, you are the type of person that makes open-source software so awesome. Thank you very much for all your time.

I think if we can populate a list of UAs to use from the GUI, then that would be absolutely stunningly perfect. Do you already have an idea of how exactly to do this?

I'd be very happy to try my best at implementing random mode. I'm not an expert at Javascript, and this is the first time I'm actually hacking on an add-on, so I am really appreciating your hand-holding!

Best! -m

[ray-lothian]

Do you already have an idea of how exactly to do this?

Just add a new button. On its call simply get the old list from chrome.storage.local.get({'random-list': []}, prefs => ...). Push the new agent and write it back withchrome.storage.local.set(obj). The background page will have access to the updated list per our discussion.

btw, since we have this new preference, we can also let it be accessible in the custom mode too. Like having a keyword that if a hostname value is set to it, then extension uses this array.

this is the first time I'm actually hacking on an add-on

No worries. Play with the code and let me know if you need any help