Linux support

[kasnder]

I've been trying to compile this on Linux and been exploring the feasibility of a port. After a couple of changes (which are pretty dirty because I've never worked with Swift), I could get the program to compile.

However, the login fails because of invalid credentials:

root@139ae75041b2:~/ipatool# .build/aarch64-unknown-linux-gnu/release/ipatool auth login --log-level debug
==> ⚠️	[Warning] Enter Apple ID email:
==> ⚠️	[Warning] Enter Apple ID password:
==> 🛠	[Debug] Creating HTTP client...
==> 🛠	[Debug] Creating App Store client...
==> ℹ️	[Info] Authenticating with the App Store...
==> 🛠	[Debug] invalidCredentials
==> ❌	[Error] Invalid credentials.

Specifically, Apple returns the following network response:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<plist version="1.0">
<dict>
<key>pings</key>
<array></array>
<key>failureType</key><string>-5000</string>
<key>customerMessage</key><string>The Apple ID you entered couldn’t be found or your password was incorrect. Please try again.</string>
<key>m-allowed</key><false/>
<key>cancel-purchase-batch</key><true/>
</dict>
</plist>

The 2FA popup worked on the first try, and then failed the second time. It doesn't seem like I got blocked because the login still works when running ipatool on my macOS.

I imagine the reason might be in different HTTP headers between macOS and Linux, maybe?

I've been using Ubuntu 20.04 on arm64 with Swift version 5.6.1, all running in Docker on macOS.

[kasnder]

My changes to build on Linux are here: https://github.com/kasnder/ipatool/tree/linux

[kasnder]

I now realised it might be related to the used MAC address.. UPDATE: Nope; doesn't with my real MAC address either.

[kasnder]

When using my credentials from my macOS on Linux for downloading, I also run into an error:

==> 🛠	[Debug] Found app: Spotify - Music and Podcasts (8.7.26).
==> 🛠	[Debug] Creating HTTP client...
==> 🛠	[Debug] Creating App Store client...
==> ℹ️	[Info] Requesting a signed copy of '324684580' from the App Store...
==> 🛠	[Debug] genericError
==> ❌	[Error] An unknown error has occurred.

Problematic App Store response:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<plist version="1.0">
<dict>
<key>pings</key>
<array></array>
<key>failureType</key><string>5002</string>
<key>customerMessage</key><string>An unknown error has occurred</string>
<key>m-allowed</key><false/>
</dict>
</plist>

[majd]

I now realised it might be related to the used MAC address.. UPDATE: Nope; doesn't with my real MAC address either.

Can you double check that you provided the MAC address in the correct format? It doesn't make sense that the HTTP requests work on one platform but not the other, especially since even the User-Agent header is spoofed.

[kasnder]

I now realised it might be related to the used MAC address.. UPDATE: Nope; doesn't with my real MAC address either.

Can you double check that you provided the MAC address in the correct format? It doesn't make sense that the HTTP requests work on one platform but not the other, especially since even the User-Agent header is spoofed.

I've double-checked the MAC address now, and its definitely not the cause of the problem.

[kasnder]

Ok, so it seems that the MAC address is actually working now. I can login, purchase and search, but get an error when trying to download:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<plist version="1.0">
<dict>
<key>pings</key>
<array></array>
<key>metrics</key>
<dict>
  <key>dialogId</key><string>MZCommerce.AuthTokenResumeFreeBuy</string>
  <key>message</key><string>Sign In to the iTunes Sto</string>
  <key>messageCode</key><string>2042</string>
  <key>options</key>
  <array>
    <string>Download</string>
    <string>Cancel</string>
  </array>
  <key>actionUrl</key><string>p25-buy.itunes.apple.com/WebObjects/MZFinance.woa/wa/volumeStoreDownloadProduct</string>
  <key>asnState</key><integer>0</integer>
  <key>eventType</key><string>dialog</string>
</dict>
<key>failureType</key><string>2042</string>
<key>customerMessage</key><string>Sign In to the iTunes Store</string>
<key>m-allowed</key><false/>
<key>dialog</key>
<dict><key>kind</key><string>authorization</string>
<key>m-allowed</key><false/>
<key>use-keychain</key><true/>
<key>message</key><string>Sign In to the iTunes Store</string>
<key>explanation</key><string>If you have an Apple ID, sign in with it here.</string>
<key>defaultButton</key><string>ok</string>
<key>okButtonString</key><string>Download</string>
<key>okButtonAction</key><dict><key>kind</key><string>Buy</string>
<key>buyParams</key><string>guid=**************&amp;salableAdamId=1586366816&amp;creditDisplay=&amp;hasBeenAuthedForBuy=true</string>
<key>itemName</key><string>null</string>
</dict>
<key>cancelButtonString</key><string>Cancel</string>
<key>initialCheckboxValue</key><true/></dict>
<key>cancel-purchase-batch</key><true/>
</dict>
</plist>

[baltpeter]

While reverse-engineering the requests for #51, I also found that I was not able to reproduce the requests on Linux. Trying to reproduce the exact headers and body of requests that worked just fine in IPATool or other tools, failed with authentication errors when I manually sent them through Insomnia or Node.js. However, I was able to reissue the requests through mitmproxy running on Linux.

My guess is that they're using TLS fingerprinting or similar. If we manage to bypass that, it shouldn't be hard to implement downloading in just about any language, the requests aren't complicated to replicate and don't depend on anything Swift-specific.

[JPS46225]

So I actually have a private functioning Python version and can confidently confirm that TLS fingerprinting isn't the issue here. That being said, it is being done on my iPad, however certificate verification has been disabled for both authentication and download requests.

[spawn9275]

However, I was able to reissue the requests through mitmproxy running on Linux.

What did you change to the request? Replaying the request on mitmproxy (with the spoofed User-Agent) will not work afaik. It gives the same error kasnder posted above.

Also, how did you guys find this endpoint: /WebObjects/MZFinance.woa/wa/volumeStoreDownloadProduct?guid= ? When I was first inspecting traffic, every download seems to be operating through /WebObjects/MZBuy.woa/wa/buyProduct on https://p26-buy.itunes.apple.com or https://p42-buy.itunes.apple.com

[spawn9275]

Update: It does work on Linux.

Here's a python script for downloading IPAs. https://gist.github.com/spawn9275/6303053b0fae58d5b777e2c6d9192a2d

The key part is the persistent session state: saved_session = requests.Session() Without it, I get the exact same errors as kasnder. Hope this helps.

[JJTech0130]

The gist seems to be gone. Did anyone save it?

[majd]

I'm working on adding official support for building ipatool on Linux. The only part that's left is finding an alternative to the macOS keychain.

[spawn9275]

The gist seems to be gone. Did anyone save it?

Whoops. Sorry, changed username. It should work now.

[kasnder]

I'm working on adding official support for building ipatool on Linux. The only part that's left is finding an alternative to the macOS keychain.

Great to hear! What's the problem with saving a configuration file in the filesystem, as I did in my fork?

That's what tools for Android app downloading, like gplaycli, do, too.

[ThomasHoooo]

Update: It does work on Linux.

Here's a python script for downloading IPAs. https://gist.github.com/spawn9275/6303053b0fae58d5b777e2c6d9192a2d

The key part is the persistent session state: saved_session = requests.Session() Without it, I get the exact same errors as kasnder. Hope this helps.

Do you have any ways to bypass the 2FA requirement?

[spawn9275]

Update: It does work on Linux. Here's a python script for downloading IPAs. https://gist.github.com/spawn9275/6303053b0fae58d5b777e2c6d9192a2d The key part is the persistent session state: saved_session = requests.Session() Without it, I get the exact same errors as kasnder. Hope this helps.

Do you have any ways to bypass the 2FA requirement?

I believe you would simply append the 2FA code to the end of the password before the second POST request. Something like this:

# First request
auth = saved_session.post(url=auth_url, headers=auth_headers, data=auth_params)    

# After the first request, you should get the 2FA code
code = input('Write 2FA code: ') 
auth_params['password'] += code

# Second request
auth = saved_session.post(url=auth_url, headers=auth_headers, data=auth_params)

[kasnder]

I'm working on adding official support for building ipatool on Linux. The only part that's left is finding an alternative to the macOS keychain.

I was wondering whether there's any update on this or any way in which I could help?

[majd]

I was wondering whether there's any update on this or any way in which I could help?

Apologies for the lack of updates. While I was porting the project to cross-compile on Linux, I encountered an issue where the calls to the auth endpoint would fail frequently. I have some suspicions that it's due to the request's GUID (MAC address with stripped :) not originating from an official Mac device, although more investigation is needed. I also did the testing only inside Docker containers, not sure if the behavior is different from other Linux installations.

[JPS46225]

I was wondering whether there's any update on this or any way in which I could help?

Apologies for the lack of updates. While I was porting the project to cross-compile on Linux, I encountered an issue where the calls to the auth endpoint would fail frequently. I have some suspicions that it's due to the request's GUID (MAC address with stripped :) not originating from an official Mac device, although more investigation is needed. I also did the testing only inside Docker containers, not sure if the behavior is different from other Linux installations.

Afaik, it doesn't even need to be from a mac, just a device associated with the user's Apple ID.

[majd]

I am considering having a required --mac-address parameter for the Linux version of the tool.

[majd]

Small update: I've been working on Linux support over the weekend and encountered an issue in the FoundationNetworking framework with Swift 5.6.2 on Linux. It seems that the URLSession does not respect the Set-Cookie headers for redirect responses. This is causing issues with the auth command.

I have two options:

1. Write custom logic to ignore the redirect requests and manually read out Set-cookie headers and update them (and hope that URLSession stores and uses them in subsequent requests properly).
2. Replace URLSession with something else that is more Linux-friendly.

[mateo-m]

Update: It does work on Linux.

Here's a python script for downloading IPAs. gist.github.com/spawn9275/6303053b0fae58d5b777e2c6d9192a2d

The key part is the persistent session state: saved_session = requests.Session() Without it, I get the exact same errors as kasnder. Hope this helps.

Hey @spawn9275, thanks for making this script ! Do you know where I should pass the password_token and store_front variables ? I want to download a(n) (free) app that I never downloaded before, so I guess this falls under the purchase use-case.

[thomas-art]

Update: It does work on Linux.

Here's a python script for downloading IPAs. https://gist.github.com/spawn9275/6303053b0fae58d5b777e2c6d9192a2d

The key part is the persistent session state: saved_session = requests.Session() Without it, I get the exact same errors as kasnder. Hope this helps.

I made a ios shortcut by reading this python script.（only for purchased apps） https://www.icloud.com/shortcuts/d6356718275145a99bef4f2ddbfadfba including 2FA support you have to replace the mac address in the shortcut first

[majd]

I am happy to finally announce that the new major version of ipatool is now available as a release candidate. It is rewritten using Go, which allowed me to add support for other operating systems such as Linux and Windows. Please do test it out on Linux and let me know if you run into any issues. Note that some commands have been updated; see below for the full list of changes.

https://github.com/majd/ipatool/releases/tag/v2.0.0-rc.1

Changes in v2.0.0-rc.1:

• Added support for Windows.
• Added support for Linux.
• Added support for generating autocompletion script using the completion command.
• Implemented new auth info command.
• Implemented --verbose flag that replaces the --debug-level flag to enable verbose logging.
• Implemented --format which allows specifying logs output format to either text or json (default: text).
• Implemented flag --non-interactive flag to disable running the tool in an interactive session.
• The relevant command (i.e. purchase) will now automatically determine the country and the device family from the authenticated account. The following flags have been deprecated.
  • --country
  • --device-family
• Improved structured logging.
• Improved error handling.
• Improved support for automated systems.
• Added unit tests to cover the majority of the private App Store API logic.

[Shonke]

@majd Can we get the mac address from the environment variable?

[majd]

@majd Can we get the mac address from the environment variable?

@Shonke What would be the use case for that?

[github-actions[bot]]

This issue has been automatically closed because there has been no response from the original author.

[adrianmihalko]

Hi,

I got this error on a headless server:

 ./ipatool-2.0.0-linux-amd64 auth login --email @gmail.com
12:13PM INF enter password:
12:13PM INF enter 2FA code:
970814
12:13PM ERR error="failed to log in: failed to save item in keychain: failed to set item in keyring: Object does not exist at path “/”" success=false

[majd]

Hi,

I got this error on a headless server:

 ./ipatool-2.0.0-linux-amd64 auth login --email @gmail.com
12:13PM INF enter password:
12:13PM INF enter 2FA code:
970814
12:13PM ERR error="failed to log in: failed to save item in keychain: failed to set item in keyring: Object does not exist at path “/”" success=false

@adrianmihalko Please do create a new issue with more details (such as what Linux distro and OS version you are using) so that this can be looked into.