mloader

mloader copied to clipboard

Like the title suggest there are free chapters and mangas which are App exclusive. Maybe you cant implement them.

Do you have an example?

The entire Chainsaw man manga is available via App.

From what I see, they are now using a different api endpoint for manga details (title_detailV2 vs title_detail used by mloader) along with some secret, we can try and utilise it.

It would be interesting if the other endpoint also lists the delisted chapters that are even hidden from the website and the app (like most of the re-editions e.g., Naruto). Some of these in between chapters seem to remain on the server and are downloadable atm. I might consider doing an 'API dump' if these are not visible.

I've made an update to use the mobile app's API. Seems to be working fine for me (and faster). If anyone wants to try it out, please install from the develop branch:

pip install git+https://github.com/hurlenko/mloader.git@develop

I'd like to hear some feedback before releasing a new version.

with naruto [title: 100018 ] you don't get the recent chapters in the app but then currently for me it only downloads up to including the 7th chapter while there are 9 in the app available and at least up to chapter 10 [chapter: 1000406 ] is on the server.

Edit: also yeah it's a lot faster for some reason

Edit Edit: the formating on spy x family [100056] for chapter 58 part 1 [1011977] and chapter 58 part 2 [1012693] does seem to not work as intended. i tried something to fix this with #38 but feel free to disregard it entirely i'm not really someone who does stuff with python

@meshstyles Thanks a lot for the feedback, will take a look at it in the next couple days, don't have much time now

there seems to be a problem with this version. The last chapter is not always correct:

with those arguments in develop branch

-l -f raw -t 100023

I'm getting

06.08.2022 20:46:39 |   INFO   |   loader.py    145  | 1/1) Manga: Rurouni Kenshin
06.08.2022 20:46:39 |   INFO   |   loader.py    146  |     Author: Nobuhiro Watsuki
06.08.2022 20:46:39 |   INFO   |   loader.py    157  |     1/1) Chapter #005: Act 5: The Fight Merchant

in master branch:

-l -r --chapter-subdir -t 100023

I'm getting

06.08.2022 20:49:00 |   INFO   |  __main__.py   206  | Started export
06.08.2022 20:49:06 |   INFO   |   loader.py    138  | 1/1) Manga: Rurouni Kenshin
06.08.2022 20:49:06 |   INFO   |   loader.py    139  |     Author: Nobuhiro Watsuki
06.08.2022 20:49:07 |   INFO   |   loader.py    150  |     1/1) Chapter #190: Act 190: Battle on Three Fronts: Round 1, Part 2
#190
06.08.2022 20:49:13 |   INFO   |  __main__.py   224  | SUCCESS

In my case I could download all 101 chainsaw man chapters at once with no struggle.

i kinda explined this poorly in my comment but it seems like the mobile app api does not have the latest chapters for some manga. but the website api does have them. This explains how there are diffrent last chapters using the dev and master branch. for example chainsaw man on the other hand has all chapters in the app but only the earliest and latest chapers on the website. So for optimal results youd need to consider both apus in some cases but i think it is not visible through the api if there are diffrences in available chapters between both apis. The only option i see here is to either request data from both api or just use a user input flag to check if the website api or the app api should be used

so there is a problem with develop branch

root@nextcloudpi:~# mloader -t 100034

            _                 _
  _ __ ___ | | ___   __ _  __| | ___ _ __
 | '_ ` _ \| |/ _ \ / _` |/ _` |/ _ \ '__|
 | | | | | | | (_) | (_| | (_| |  __/ |
 |_| |_| |_|_|\___/ \__,_|\__,_|\___|_|

04.09.2022 13:29:32 |   INFO   |  __main__.py   202  | Started export
04.09.2022 13:29:33 |   INFO   |   loader.py    145  | 1/1) Manga:
04.09.2022 13:29:33 |   INFO   |   loader.py    146  |     Author:
04.09.2022 13:29:33 |   INFO   |  __main__.py   219  | SUCCESS

After updating AUTH_PARAMS obtained from my android phone, everything works, so unfortunately the "secret" value has a certain expiration time

How can I fix this?

I'm new to this command-line thing and want to read chainsaw man without having to deal with the "only avaible to read one time" bs of manga-plus

so there is a problem with develop branch

root@nextcloudpi:~# mloader -t 100034

            _                 _
  _ __ ___ | | ___   __ _  __| | ___ _ __
 | '_ ` _ \| |/ _ \ / _` |/ _` |/ _ \ '__|
 | | | | | | | (_) | (_| | (_| |  __/ |
 |_| |_| |_|_|\___/ \__,_|\__,_|\___|_|

04.09.2022 13:29:32 |   INFO   |  __main__.py   202  | Started export
04.09.2022 13:29:33 |   INFO   |   loader.py    145  | 1/1) Manga:
04.09.2022 13:29:33 |   INFO   |   loader.py    146  |     Author:
04.09.2022 13:29:33 |   INFO   |  __main__.py   219  | SUCCESS

After updating AUTH_PARAMS obtained from my android phone, everything works, so unfortunately the "secret" value has a certain expiration time

I'm actually shocked but all you need to do is change the app version. seems like the secret is valid for some time but yeah it looks like some continous updates buming the version number are required for mobile.

How can I download that Free chapters from the App? I need to change the secret value?

It would be nice if @janowakps or @hurlenko write how to get the secret key in the app and set it in mloader.

How can I download that Free chapters from the App? I need to change the secret value?

you need the development branch the feature is yet to be made public

It would be nice if @janowakps or @hurlenko write how to get the secret key in the app and set it in mloader.

idk if that would m+ would like that tbh

So for everybody coming to this thread and thinking about how to get the secret key:

• you need to run the app (apk-file) in an emulator or in my case on mac with android sdk.
• copy the app data from /data/data/{M+ APP directory}/shared_prefs/config.xml
• in the config.xml you will find the secret
• copy the secret into the loader.py from mloader

@hurlenko your develop branch is working like a charm. The download order is sometimes a bit weird but works completely fine

Same problem as @IsaBNekomata and @meshstyles, changing the secret code didn't change anything.

@IsaBNekomata and @meshstyles you just need to update version number (check the app store to get it) in loader.py. You need the version from development branch to even have the option though.

@hurlenko As you probably know Shueisha locked some series behind a paywall of subscription. These series only have 3 first chapters available and after downloading them, the script throws an error. Assassination Clasroom (100001) is an example. Would be great if the error could throw some human readable message. I'm also curious - if I were to pay for subscription, would it be recognized by the script and let me download the completed series? Does secret key carry that information?

@hydreg didn't know about paywall. The script has no authorization so it knows nothing about you paying for subscription. I've opened two issues (#50 and #51) but I can't give the estimates on when I might get to them.

@hurlenko Take your time, if it ever works out we'll be able to get complete archives anyways.

@IsaBNekomata and @meshstyles you just need to update version number (check the app store to get it) in loader.py. You need the version from development branch to even have the option though.

I think i did a rather poor job at explining what i ment with the app version number. to make it easier for anyone not in the know mloader with app api support was probaly bulilt looking at the ios app. so when the ios app has an update you need to change the app version in loader.py AUTH_PARAMS to a current version of ios app. Tbh this seems like it'd be a chore to keep up with so i think it might be best to have a cli param that could so people don't need to install from source if it takes a while for an update through pip.

@hydreg didn't know about paywall. The script has no authorization so it knows nothing about you paying for subscription. I've opened two issues (#50 and #51) but I can't give the estimates on when I might get to them.

I think they've introduced that at the start of the month so it's still kinda new. But the highest tier allows you access to basically everything they had.

@hurlenko Take your time, if it ever works out we'll be able to get complete archives anyways. I'll be honest some manga have poor quality so... yeah idk if this is the approach I'd recomend other then to have fun or learn how automate or datamine.

One issue I discovered why attempting to use the devel branch was that it did not detect the .5 chapters. Example would be for Hokkaido Gals Are Super Adorable! (100116) which it found chapters 100 and 101, but not 100.5

I can from Desktop Browser that they are using https://jumpg-webapi.tokyo-cdn.com/api/title_detailV3?title_id=xxxx for the api. Rest addresses are still the same.

Another note: Even if the manga page from website shows more than 6 chapters free to read on desktop without subscription, mloader can only get the oldest 3 + latest 3 chapters 😢 Any way to fix that?

Hmm, I updated the AUTH_PARAMS values from my Phone's App and it's currently downloading All chapters. 😄

Any ideas if BlueStacks can be used? I rooted it by modifying bluestacks.conf, then found the config.xml and got the secret, but couldn't get it to work.

Here's the config I put in loader.py:

AUTH_PARAMS = {
    "app_ver": "1.9.19",
    "os": "android",
    "os_ver": "7.1.2",
    "secret": "[redacted]",
}

Here's the traceback:

28.01.2024 20:43:22 |   INFO   |  __main__.py   206  | Started export
28.01.2024 20:43:23 |   INFO   |   loader.py    140  | 1/2) Manga: 
28.01.2024 20:43:23 |   INFO   |   loader.py    141  |     Author: 
28.01.2024 20:43:23 |  ERROR   |  __main__.py   226  | Failed to download manga
Traceback (most recent call last):
  File "[redacted]\venv\Lib\site-packages\mloader\__main__.py", line 218, in main
    loader.download(
  File "[redacted]\venv\Lib\site-packages\mloader\loader.py", line 191, in download
    self._download(manga_list)
  File "[redacted]\venv\Lib\site-packages\mloader\loader.py", line 146, in _download
    chapter = viewer.pages[-1].last_page.current_chapter
              ~~~~~~~~~~~~^^^^
  File "[redacted]\venv\Lib\site-packages\google\protobuf\internal\containers.py", line 93, in __getitem__
    return self._values[key]
           ~~~~~~~~~~~~^^^^^
IndexError: list index out of range
28.01.2024 20:43:23 |   INFO   |  __main__.py   227  | SUCCESS

I'm currently on BlueStacks 5.8 if that makes a difference.

@10SecondsToWaste, when the error says IndexError: list index out of range, it's mostly because the manga you want to download needs MangaPlus MAX subscription (Premium). Some other cause is also possible. Try different manga and see what happens.

@rokibhasansagar

when the error says IndexError: list index out of range, it's mostly because the manga you want to download needs MangaPlus MAX subscription (Premium)

I don't think so. The chapters I want to download are available for one-time reading and I have not read any of them since I downloaded the app.

Try different manga and see what happens.

Same error. Not even the latest chapters can be downloaded. Here's what the commands I used look like:

$ python -m mloader --out ./foo --format raw --chapter 1019618

I got the chapter IDs from old Reddit posts in each manga's subreddit (for Two On Ice it's r/TwoOnIce, etc.) so they are definitely correct.

I got the chapter IDs from old Reddit posts in each manga's subreddit (for Two On Ice it's r/TwoOnIce, etc.) so they are definitely correct.

Does it download correctly if you use ids you can access via browser, for example 1019569? I'm almost certain the id you've provided has a typo and should be 1019608.