Custom circuit-builder

[david415]

Aaron Gibson of Tor Project suggested we add a feature allowing the user of txtorcon to specify custom circuit paths. He's written some rough draft code to do this for ooni-probe... but he suggested we should take it as inspiration and write a clean feature for txtorcon to do this. He pointed me to his code here:

https://github.com/TheTorProject/ooni-probe/blob/feature/tor_test_template/ooni/utils/tor.py

Update: please read the end first, much of the "upper stuff" is already implemented

[meejah]

txtorcon already does this, see https://github.com/meejah/txtorcon/blob/master/examples/circuit_for_next_stream.py or the attach-by-country example both of which build custom circuits and then attach streams to them.

Also, "carml" can do it from the command-line (carml circ --build *,*,exithash for example or carml circ --build auto).

[meejah]

Hmm, maybe you mean something higher-level like exists in TorFlow? I once tried to understand what that code was really doing, but failed.

But at a high level it allowed you to specify "classes" of circuits, not as low-level as choosing specific relays etc. Is that what you really mean for this bug-report?

That is, you could say "I want a top-10-percentile Guard, an average middle, and one of these 5 exits" and get a circuit-generator back or something? This does sound like a useful feature to me, but I don't know what some "real" use-cases are.

For anyone looking at this, another source of "interesting" code is the exit-scanner branch in this repository -- it's out-of-date but was used to produce some metrics around "average" failed-circuit rates.

[meejah]

Worth noting I think: it's not possible to attach streams destined to hidden-services (even if the circuit is a rendevous circuit etc); this would require a patch to tor.

[aagbsn]

This feature branch is a bit more up to date: https://github.com/TheTorProject/ooni-probe/blob/feature/stream_bw/ooni/utils/tor.py

My use case requires that I can build multiple circuits that carry streams to the same destination, over different paths. I do this with the OnionRoutedTCPClientEndpoint, TorCircuitContextFactory, StreamAttacher, AttacherDispatcher classes. Changing path selection is as easy as subclassing StreamAttacher and implementing request_circuit_build. It's true that the examples provide a starting point, but I think that txtorcon should provide a API for specifying path selection at stream granularity. My code works - but I think it could probably be improved substantially - and it would be more useful to other developers if these features were available from txtorcon directly.

How this works: In order to correctly map new streams to a specific circuit, the StreamAttacher must distinguish between streams, but as streams may have same destination, you need to use the source address and port to differentiate between streams. My OnionRoutedTCPEndpoint wraps the SOCKS5ClientFactory to learn the source ip, port before tor creates the stream. When the stream is created, the new stream event contains the stream_id and source ip, port that is used to match the stream id with the endpoint. The OnionRoutedTCPClientEndpoint's TorCircuitContextFactory describes which StreamAttacher instance should be used.

The second feature is a way to easily plug in different StreamAttachers, simultaneously. This is done by a singleton proxy class AttacherDispatcher that proxies calls to attach_stream to a StreamAttacher instance. This is a workaround as txtorcon currently only supports a single stream attacher. This could be better engineered and implemented in txtorcon directly.

[meejah]

Yes, I agree about multiple attachers -- but care would be needed for ordering etc. (e.g. call them in order? do they chain?) but not that hard.

As to the stream/circuit stuff: this does sound valuable (basically, "higher level combination of .build_circuit and IStreamAttacher"). So it sounds like basically you want to say "give me a Tor stream, and by the way use this path".

So at the simplest you'd want something like OnionRoutedTCPClientEndpoint (or TorClientEndpoint as it exists currently in txtorcon) to take some parameters indicating a circuit/path to use?

So these parameters could conceivably be:

• a Circuit instance
• a list of Tor relay IDs
• ? some kind of selection criteria/function that returns either of the above

I think the most-applicable to your use-case is probably the list-of-relay-IDs?

Thinking a little wider, maybe there's really two major sub-features here: "a thing which builds circuits somehow" and "a TorClientEndpoint that routes its stream over a specific circuit". Then, those can be composed to provide the "thing which builds and reuses circuits to put streams on" (which is the thing you really want).

[meejah]

This could potentially be exposed to the endpoint string-parsers, too, so you could maybe have tor:host=torproject.org:port=443:path=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa,bbbbbb.... but that might not be a good idea...

[meejah]

What if Circuit instances provided a .create_client_endpoint method. This would return an IStreamClientEndpoint implementation that -- when .connect() is called on it -- makes sure the resulting stream goes over the original Circuit.

Thus, you could immediately make a client-type connection over any Circuit instance you have.

So, for example, if you have a TorState instance in state, something like:

circuit = yield state.build_circuit([relay0, relay1, relay2])
ep = circuit.create_client_endpoint()
proto = yield ep.connect(MyProtocolFactory)

Presumably tying this together with twisted.web.client.Agent would be desirable, too...

[meejah]

Looking at the code @aagbsn pointed to, it seems like the uses-case that code is interested in is "a stream over a circuit which has a particular exit" (doesn't care about entry, middle).

I think adding create_client_endpoint would work well for this case, and splits up the "create circuits" code from "connect streams" -- but still makes it easy to tie them together. So this method would return an object like OnionRoutedTCPClientEndpoint in the code linked to in the original issue.

This would also probably benefit from a higher-level circuit-building object (which would be along the lines of what the "MetaTroller" stuff in TorFlow does). That is, configuring "types" of circuits and getting a certain number of them built. So, e.g., "build circuits which have exit-node X". Obviously, these would build on top of TorState.build_circuit.

[meejah]

This feature-request should be re-visited in light of the new circuit-attaching APIs Circuit.stream_via, meaning no "user" could should need to mess around with IStreamAttacher at all any more. The only "user code" stuff to do is to build the desired Circuit instances.

[ghost]

Hi! i just visited this issue and i want to share some experience: I was working on a browser bundle with a js-addon installed on it for controlling the browser traffic through tor. -> by controlling i mean: deciding what criteria must circuit's path consider. these criteria's were: choosing countries for entry, middle and exit position; and by middle position, i mean you can specify any number of country, and they will be used for each hop, country-code in the middle of the path as well as entry and exit. for instance: entry->US middle -> IR,CH,FR exit -> RO controller will create path's with the length of 5 and each hop, is from the specified country.

for this i tried to hack into OPADDON from torflow project. the problem i faced was this -> i wanted to isolate my tabs, but no api was available; I looked on stem, they offered user/pass on their CircuitEvent api, but i needed this on the StreamEvent api, so i can tell that these bunch of streams, are from tab A, and these other streams are from tab B, streams of tab A --ATTACH--> circ A. and streams of tab B -->ATTACH--> circ B. now on this issue i see some very good considerations about this problem, when you guys mentioned SOCKS5ClientFactory, combining this with a customized IStreamAttacher to choose the circuit for streams, is a good progress for the problem i mentioned. Thanks in advance. By the way, how should i be in touch for the Build Many Circuits feature you mentioned in the guide?

[meejah]

hi @pouyamiralayi I didn't see this before now, sorry! You might like to look at carml and the carml stream --attach feature. See https://carml.readthedocs.io/en/latest/

[meejah]

I think the remaining part to make here is "a thing which creates circuits, according to some criteria". The use-cases about easily attaching streams to circuits are covered by stream_via