Any plan of Android support ?

[jkyeo]

[NghiaTranUIT]

I actually dig around Android world to find the equivalent function of Method Swizzling, but I couldn't find any useful.

It seems that Kotlin doesn't support it, and Java have Reflection, but it doesn't seem what I'm looking for.

Do you have any suggestion or code example 🤔 @jkyeo

[djbe]

Very curious about this as well (just in general, for all my Android colleagues).

From some quick searching, these seem to be the current solutions:

1. Android Studio has a built-in Network Profiler
2. There are libraries for in-app network debugging such as Chuck, but they require the user making all calls via specially configured OkHttp clients (by adding an interceptor).
3. Then there are Android tools such as HttpCanary which just do the VPN (or proxy) dance to intercept all traffic (like Charles does).

For something like Atlantis or Bagel to work, the libraries need to be able to monitor all calls made by the system (in the app), either by using some built-in mechanism, or by swizzling (replacing the system implementations).

A possible solution may be option 2, by having the developer add interceptors to each "client" they want to monitor, something like Atlantis.registerClient(...).

[NghiaTranUIT]

Thanks for the detailed hint @djbe. I will continue researching and soon support Atlantis on Android 👍

[aminerol]

in Android, they have what they call dynamic instrumentation where you can set up hooks on methods. not sure if this is what it's needed or the equivalent for Method Swizzling in android but there are many libraries abstracting this concept such as AndHook

[djbe]

Right, so a colleague of mine made an Android implementation for Bagel, credits to @dieter115!

Right now the code is embedded in an internal library we use in our Android projects. It consists of:

• Interceptor and listener autodiscovery, code can be found here. It's very similar to the iOS code.
• The interceptor is added to the client(s) the developer wants to monitor: interceptor creation and registration
• The service discovery is started on app onCreate() using BagelNetworkDiscoveryManager.registerService(context), the equivalent of Atlantis.start()

Now we'd love to switch to Proxyman for our Android devs (using Atlantis). We can probably tweak our implementation to match the Atlantis protocol, or would you rather do this yourself?

[NghiaTranUIT]

@djbe It sounds good. I'm not familiar with Android development, so it might take more time to do it by myself 😸

If you don't mind, you can do and open the PR. I'm happy to help you with the integration 😄

[djbe]

Is there a high level documentation on how Atlantis works, and maybe where it differs from Bagel? I noticed Atlantis for example compresses all sent data using gzip

[NghiaTranUIT]

@djbe Unfortunately, there is no high-level documentation, but I can quickly describe here:

• As soon as you launch Atlantis, Atlantis will do "Method Swizzling" for all NSURLSession methods: It's the same logic with Bagel, but in Swift. Code: https://github.com/ProxymanApp/atlantis/blob/main/Sources/NetworkInjector.swift
• When Atlantis receives the request/response. It will construct the message (https://github.com/ProxymanApp/atlantis/blob/89702b9586fcbed2e99a5e896821461bc700272c/Sources/Atlantis.swift#L298)
• Then, sending it to macOS via Bonjour Service (https://github.com/ProxymanApp/atlantis/blob/89702b9586fcbed2e99a5e896821461bc700272c/Sources/Transporter.swift#L104)
• gzip is optioned, but recommend. Proxyman for macOS will check if it's a gzip or not.

---

If you're going to implement the Android, you can implement your own "Method Swizzling" in Android, then construct a message with the given Request/Response and pass it to Atlantis with this func (https://github.com/ProxymanApp/atlantis#1-my-app-uses-c-network-library-and-doesnt-use-urlsession-nsurlsession-or-any-ios-networking-library)

In this way, Atlantis will handle the rest (Construct the message and send it to Proxyman for macOS) 👍

[dieter115]

Hey Guys I have been working on a POC for Android. And I made it possible to connect my application (Android phone) to a Proxyman socket on my macbook. I wrote your message ,device , project , ... objects in kotlin and I am trying to send this data gzipped json or normal json over this socket. I used Wireshark to check the packages and connection and they were all sent to the right socket. They don't seem to appear in the Proxyman app ... Is there a format the message and / or it's fields need to have to be recognized by Proxyman.

I tried it with escape slashes

{"buildVersion":"1.0-dev","content":"{\"device\":{\"model\":\"OnePlus,IN2023; Android/30\",\"name\":\"OnePlus 8 Pro\"},\"project\":{\"bundleIdentifier\":\"be.dietervaesen.bageltester.dev\",\"name\":\"ProxyManPOC\"}}","id":"be.dietervaesen.bageltester.dev-OnePlus,IN2023; Android/30","messageType":"connection"}

and without

{"buildVersion":"1.0-dev","content":{"device":{"model":"OnePlus,IN2023; Android/30","name":"OnePlus 8 Pro"},"project":{"bundleIdentifier":"be.dietervaesen.bageltester.dev","name":"ProxyManPOC"}},"id":"be.dietervaesen.bageltester.dev-OnePlus,IN2023; Android/30","messageType":"connection"}

[NghiaTranUIT]

Hey @dieter115, If you connect to Proxyman Socket at localhost:9090, and send the Atlantis data, it won't work.

You might have someway to consume Bonjour Service from Proxyman app. From what I googled, there is Network Service Discovery from Android, which is allowed you to discover Proxyman and send the data to. Ref: https://jaanus.com/implementing-bonjour-across-ios-and-android/

Here is the Bonjour Service from Proxyman app

        static let netServiceDomain = ""
        static let netServiceType = "_Proxyman._tcp"
        static let netServiceName = "Proxyman"
        static let netServicePort: Int32 = 10909

let uniqueServiceName = "\(netServiceName)-\(UUID().uuidString)"
let netService = NetService(domain: netServiceDomain, type: netServiceType, name: uniqueServiceName, port: netServicePort)

---

From your JSON, the second one (without escape) looks correct. Basically, it's a JSON, we don't need to escape it.

However, in order to send the message to Proxyman, you might construct the message like Atlantis does. Code: https://github.com/ProxymanApp/atlantis/blob/87107de8c2881dd86b6b53da6519a93048f40ed3/Sources/Transporter.swift#L115-L118

The message will contain two parts:

1. Reserve xx bytes (Int(MemoryLayout<UInt64>.stride)) and fill it will the length of the JSON message
2. The actual JSON message in byte.

The reason is that Proxyman doesn't know when the message is completed (The message can be spliced into small chunks when sending in TCP layer). Thus, the first part is important to know where the JSON Message is done.

[dieter115]

hey @NghiaTranUIT I'm indeed connected with the service with port 10909 and I found it through Network Service Discovery. I already used this part in my Bagel POC. I create a socket with its IP and port and send packages through it. I'm already sending the length of the json data and then the json itself each as its own package/message. But what you mean is that you send 1 message with length of data and the date itself combined as 1 package?

[NghiaTranUIT]

@dieter115 you can see the code how the message is constructed https://github.com/ProxymanApp/atlantis/blob/87107de8c2881dd86b6b53da6519a93048f40ed3/Sources/Transporter.swift#L115-L118

Yes, it’s one message that consists of two parts.

[NghiaTranUIT]

I think that you can roughly translate to Kotlin. Please let me know if you need some help 😊

[dieter115]

Hey @NghiaTranUIT I reworked my code so it sends 1 message instead of 2 different messages. I used this code :

1. Sending size of message as a byte array

 val outPutStream = socket.getOutputStream()
                        val messageByteArray =Gson().toJson(proxymanRequestMessage).toByteArray()

                        val output = ByteArrayOutputStream()
                        output.write(messageByteArray.size.toLong().toByteArray())
                        output.write(messageByteArray)
                        outPutStream.write(output.toByteArray())

This sends a message that looks like this I think this is also how Atlantis does it ? But I also tried other formats for the size to see if they would work.

2. Sending size of message as a string or int

...
output.write(messageByteArray.size.toString().toByteArray())
...

or

...
output.write(messageByteArray.size)
...

These seem to make Proxyman stop.

[NghiaTranUIT]

buffer.append(&lengthPackage, length: Int(MemoryLayout<UInt64>.stride)) 

@dieter115 It means the buffer will reserve 8 bytes to store the lengthPackage, which is in bytes too.

Here is how Proxyman app get the Atlantis message

1. First of all, Proxyman will read 8 bytes on the stream and parse it to UInt64 to know the message length

        let byLength = Int(MemoryLayout<UInt64>.stride)
        connection.receive(minimumIncompleteLength: byLength, maximumLength: byLength) {[weak self] (data, _, _, error) in
            guard let strongSelf = self else { return }

            if let data = data {
                // Cast byte to UInt64
                var length: UInt64 = 0
                _ = withUnsafeMutablePointer(to: &length) { (length) -> Int in
                    data.copyBytes(to: UnsafeMutableBufferPointer(start: length, count: MemoryLayout<UInt64>.stride))
                }
...

2. After getting the length, Proxyman tries to read the message

connection.receive(minimumIncompleteLength: length, maximumLength: length)

---

From what I see in your code

output.write(messageByteArray.size.toString().toByteArray())
output.write(messageByteArray.size)

It looks like your write a length without reserving 8 bytes. and it must be an Integer or UInt64

---

It's similar to how Bagel works https://github.com/yagiz/Bagel/blob/763aadbf6a099681a66e69522f902e33094f6a55/iOS/Source/BagelBrowser.m#L124-L126

So if you code is already working with Bagel, it should work with Proxyman app

[dieter115]

@NghiaTranUIT The first approach I tried was using the same code as with Bagel. But here I send 2 messages over the socket 1 with length of message and 1 with the message data. It works with Bagel but Proxyman doesn't seem to show my sent data.. Like you can see in the picture of the message I sent.. The length of the message reservers / takes 8 bytes.

So like Bagel

val outPutStream = socket.getOutputStream()
                        val messageByteArray =Gson().toJson(proxymanRequestMessage).toByteArray()

                   
                        //first sent length of message to bagel
                         outPutStream.write(messageByteArray.size.toLong().toByteArray())
                        //sent actual message as a bytearray
                        outPutStream.write(messageByteArray)


                        outPutStream.flush()

Force it as 1 message like Atlantis https://github.com/ProxymanApp/atlantis/issues/28#issuecomment-822344148

Or is it possible that the app (buildConnectionMessage) only shows up in Proxyman after you sent a Traffic message for that app? Is there a way to debug if Proxyman is receiving anything?

[djbe]

Quick update: @dieter115 and I were able to get a PoC working with Proxyman today, the issue had nothing to do with the message size / byte reservation.

After using Wireshark and temporarily disabling gzip compression in Atlantis, we found the issue: as a side effect of using Codable, it'll automatically base64 encode any Data property. This is something that Proxyman does that's quite different from Bagel. Besides that, we found some other small differences, but that was the main one (it applies to content, requestBody, responseBody, etc...).

We found another difference, where I have no idea why Proxyman changed this from Bagel:

• Bagel sends a message at the start of a request (with only the request content), and a second message at the end of a request with the full info (request, response, etc...)
• Proxyman only expects a message at the end of a request.

If we send traffic messages with the same ID to Proxyman like we did with Bagel, it'll appear twice in Proxyman (see screenshot). I've created a separate issue for this, as it would be a general improvement to handle this.

[NghiaTranUIT]

Thanks for your input @djbe

1. as a side effect of using Codable, it'll automatically base64 encode any Data property.

Yes, it's. Codable automatically encoded the raw data to base64. I intentionally use it for easier implementation.

2. Bagel sends a message at the start of a request (with only the request content), and a second message at the end of a request with the full info (request, response, etc...)

You're right. It's the main difference between Bagel and Proxyman.

Instead of sending two parts (Request and Response) of the 1 request, Atlantis accumulates it. and send ONCE when it's done. It increases the performance of your app has a large number of traffic that sending to Proxyman.

Here is the function that you can manually send a request and response to Proxyman: https://github.com/ProxymanApp/atlantis/blob/5ca79443ec6d68b6fa5b6b76b8ddf3070e69b45a/Sources/Atlantis%2BManual.swift#L18-L30

[dieter115]

@NghiaTranUIT Me and my colleague David made it possible to send packages to Proxyman. But I still have a question.. I was making the feature where you can limit the Mac's you connect with. I copied this from my Bagel implementation but then I saw a difference. When I find bagel in the network discovery code it gives me a name like "macbook Dieter" but when i discover Proxyman services I get names like Proxyman-DA9E3C12-74DC-4865-9690-AA55FD351690 did you guys ever encounter something similar ?

[djbe]

Right so the difference with Bagel is that Bagel sets the service name to the hostname, whereas Proxyman sets the service name to some random identifier (like @dieter115 posted).

We've tried switching to using InetAddress' getHostName() and getCanonicalHostName(), but neither work well. It usually just returns the host's IP address instead of a nice network name. I'm not sure there's any solution to this, Android's reverse DNS lookup seems to be quite buggy.

[NghiaTranUIT]

Glad to know you finally send a request to Proxyman 🥇 @djbe @djbe 👍

You're right about the Proxyman Service Name, which is Proxyman-uuid. I intentionally added the suffix to prevent NSNetServicesCollisionError if we have multiple Proxyman apps in the same network.

Please try this beta build, I improve the Service Name to Proxyman-<host_name> as Bagel does. (I believe that hostname is unique for each machines in the same network, so there is no Collision Error anymore)

https://proxyman.s3.us-east-2.amazonaws.com/beta/Proxyman_2.26.0_Better_Bonjour_Service_Name.dmg

[djbe]

Quick update: tried that beta build, and got this as a service name:

Proxyman-ptr-4xa9d8d5k2butaudew8.18120a2.ip6.access.telenet.be

Whereas the laptop's name is MacBook-Pro-van-dieter-2, so no idea where it's getting that name. At first glance seems like a combination of random identifier, ipv6 address, and ISP (telenet).

[NghiaTranUIT]

@djbe can you clarify how you get this name? 🤔 It'd be great if we have a sample code that I can test out

I tested with Discovery app, which lists all available Bonjour services.

• In the latest build, the hostname has a UUID as a suffix

• In the beta build, it has a nicer name

[dieter115]

hey @NghiaTranUIT it depends on the network it seems ! So you were right !! On the network in the office I get right hostname

at home I get this

[djbe]

What @dieter115 forgot to mention was that the device name / hostname / ... when going to "Certificates > iOS > Atlantis" is correct. Is Proxyman using different code to set that service name (in the beta build) compared to that view?

[NghiaTranUIT]

In the beta build, I use Host().current.name, which returns a Host/Computer name in the network.

It's the same result when executing the following code on the Terminal app

$ hostname
Nghias-Mac-mini.local

Ref: https://developer.apple.com/documentation/foundation/host/1416949-name

---

I suppose that the hostname might be changed when you're using a corporate network? 🤔

[messi]

I am excited to read that @dieter115 and @djbe found a way to bring Atlantis to Android. What's the progress on this work? Is there any plan to make this official? And if not, is there any way to profit of that work and get some code snippets? :)

[sjmueller]

We're also interested in Android support. What's really confusing is that the Proxyman repo README advertises Android, yet it is absent in Atlantis?

[NghiaTranUIT]

@sjmueller Atlants is written by Swift, so it only supports iOS, macOS, tvOS, watchOS.

Regarding the readme, it means Proxyman can capture from Android Devices (Without using the Atlantis Framework). Please check out this doc: https://docs.proxyman.io/debug-devices/android-device

For the Android Emulator, Proxyman also provides the automatic script to do the boring part for you (Override HTTP Proxy and install the certificate), ref: https://docs.proxyman.io/debug-devices/android-device/automatic-script-for-android-emulator

[dieter115]

@messi Sorry for the late reply but we made it work and are using it in our Android projects. At the moment I haven't found a way to catch all the network traffic but I made something that works with Retrofit and Okhttp. It catches all the api calls and sends the requests/responses to Proxyman. If you want I can put in on a seperate repo? If thats ok for @NghiaTranUIT .