how to config cold start class and cold start method?

[ly20050516]

how to config cold start class and cold start method? Could someone give a example about them?Thanks

[justinjhendrick]

I'm not sure what you mean. What exactly do you want to configure?

[BoldShwan]

I have a same question.

The article "Open-sourcing ReDex: Making Android apps smaller and faster" (https://code.facebook.com/posts/998080480282805/open-sourcing-redex-making-android-apps-smaller-and-faster/), in 'Feedback-directed class layout' paragraph, is describing "We then feed this class trace into ReDex, which places the classes used during cold start first in the dex."

So, I wonder how to feed the class trace into ReDex. I thought the ~.config file has those list of classes to feed but it's not. Would you please provide an example of those classes list and how to give it to ReDex? I already have the list of the classes but I don't know how to feed them to ReDex.

Thank you.

[justinjhendrick]

Sure! This page gives some more details about the whole process.

But, to answer your specific question, see this section

[RCoekaerts]

I don't seem to get the InterDexPass to work. I suspect I might put the option "coldstart_classes" or the list_of_classes file in the wrong place

When I add "InterDexPass" together with the option -"coldstart_classes": "list_of_classes.txt" to the config file, the produced dexfile doesn't seem to change compared to the produced dexfile without these options enabled.

I managed to extract the list of classes with the python tool and placed it in both the config directory and the regex-root directory. I added the option "coldstart_classes" to the general configuration options, but when that didn't change anything I also tried variations like "InterDexPass" : { "coldstart_classes": "list_of_classes.txt" }, or "redex": { "passes": [ .... ( all passes here) ], "cold_start_classes" : "list_of_classes.txt" } but all this still didn't change anything. The problem is that redex doesn't throw any errors to indicate what I am doing wrong. I know that the config-file is read, as messing up the JSON in that file will throw an error.

Could you clarify what I am doing wrong? I have included my config file to show what I originally thought would be the right approach. (I had to add .txt to be able to drop it in here) interdex.config.txt

[RCoekaerts]

As a bit of a unrelated question, I was also wondering where the default redex command gets its configuration? I noticed that redex source.apk -o destination.apk and redex source.apk -c default.config -o destination.apk produce different classes.dex files, while I originally thought default.config would be the default configuration.

[justinjhendrick]

Sorry for the delay. I just got back from the holidays.

Let's get some more logging. Could you set an environment variable, like this:

export TRACE=1,IDEX:4

(that might be too verbose, you can decrease the IDEX:4 number to get less log spew)

I suspect that you'll see this message:

InterDexPass not run because no ProGuard configuration was provided.

We usually run proguard before redex (and even if we don't, we still parse a proguard config file). So, if you have a proguard config file, pass it to redex with -P config.pro. If you don't have a config, an empty one might do the trick.

And about the redex config. Yeah, that's confusing. When you don't specify -c, a very old config is baked into the source code: https://github.com/facebook/redex/blob/stable/tools/redex-all/main.cpp#L150. We plan to fix that soon. default.config is more up to date (but also more aggressive).

[RCoekaerts]

Hi Justin,

I added the logging and I indeed got the message you mentioned. So I added a proguard configuration file with the -P option but that didn't solve the problem. I still get the message: InterDexPass not run because no ProGuard configuration was provided. However when I misspel the proguard configuration filename, redex crashes, so that indicates to me the file is indeed found. Among the log output is also the following: Parsed ProGuard config file completed in 0.0 seconds I tried an empty file and a file with multiple proguard statements in it, but the message always remains.

Do you have any insight what I could be doing wrong?

[justinjhendrick]

hmm...

Try making one of those proguard statements a "-keep" rule. It looks like this line: https://github.com/facebook/redex/blob/d009f4e8e2e19200af8bedf4651b371d15bdd8cd/libredex/PassManager.h#L56-L58 looks explicitly for keep rules, though I'm not sure why.

[RCoekaerts]

I have keep rules in my proguard configuration. That doesn't change the outcome.

[justinjhendrick]

Huh.

Could you post your proguard config and the output of redex (with TRACE on)?

Another thing we can try is attach a debugger and put a breakpoint here to see if it's called.

https://github.com/facebook/redex/blob/d009f4e8e2e19200af8bedf4651b371d15bdd8cd/libredex/ProguardParser.cpp#L705

[RCoekaerts]

config.txt

It's a simple file that I just made to test it out. I changed the extension from .pro to .txt to upload it here.

[RCoekaerts]

Forgot the redex output:

Using config config/interdex.config Using binary (default) Extracting apk... Detected dex mode Api21DexMode Unpacking dex files Detecting Application Modules Unpacking APK finished in 0.04 seconds Running redex-all on 1 dex files Running redex binary at /tmp/redex.hL1fVg/redex-all Parsed ProGuard config file completed in 0.0 seconds Loading classes from dex from /tmp/redex.hL1fVg/tmpiu9glax9.redex_dexen/dex0/classes.dex Load classes from dexes completed in 0.3 seconds Deobfuscating dex elements completed in 0.0 seconds Initializing reachable classes completed in 0.6 seconds Processing proguard rules completed in 0.0 seconds Evaluating ReBindRefsPass... ReBindRefsPass (eval) completed in 0.0 seconds Evaluating BridgePass... BridgePass (eval) completed in 0.0 seconds Evaluating SynthPass... SynthPass (eval) completed in 0.0 seconds Evaluating FinalInlinePass... FinalInlinePass (eval) completed in 0.0 seconds Evaluating DelSuperPass... DelSuperPass (eval) completed in 0.0 seconds Evaluating SimpleInlinePass... SimpleInlinePass (eval) completed in 0.0 seconds Evaluating PeepholePass... PeepholePass (eval) completed in 0.0 seconds Evaluating ConstantPropagationPass... ConstantPropagationPass (eval) completed in 0.0 seconds Evaluating LocalDcePass... LocalDcePass (eval) completed in 0.0 seconds Evaluating RemoveUnreachablePass... RemoveUnreachablePass (eval) completed in 0.0 seconds Evaluating RemoveGotosPass... RemoveGotosPass (eval) completed in 0.0 seconds Evaluating DedupBlocksPass... DedupBlocksPass (eval) completed in 0.0 seconds Evaluating SingleImplPass... SingleImplPass (eval) completed in 0.0 seconds Evaluating ReorderInterfacesPass... ReorderInterfacesPass (eval) completed in 0.0 seconds Evaluating RemoveEmptyClassesPass... RemoveEmptyClassesPass (eval) completed in 0.0 seconds Evaluating ShortenSrcStringsPass... ShortenSrcStringsPass (eval) completed in 0.0 seconds Evaluating RegAllocPass... RegAllocPass (eval) completed in 0.0 seconds Evaluating CopyPropagationPass... CopyPropagationPass (eval) completed in 0.0 seconds Evaluating LocalDcePass... LocalDcePass (eval) completed in 0.0 seconds Evaluating InterDexPass... InterDexPass (eval) completed in 0.0 seconds Running ReBindRefsPass... field_refs [call sites: 2623, old refs: 561, new refs: 241] method_refs [call sites: 2061, old refs: 1081, new refs: 559] array_clone [call sites: 34, old refs: 34, new refs: 1] equals [call sites: 152, old refs: 8, new refs: 1] hashCode [call sites: 67, old refs: 6, new refs: 1] getClass [call sites: 72, old refs: 1, new refs: 1] ReBindRefsPass (run) completed in 0.0 seconds Running BridgePass... BridgePass not run because no ProGuard configuration was provided. BridgePass (run) completed in 0.0 seconds Running SynthPass... SynthPass not run because no ProGuard configuration was provided. SynthPass (run) completed in 0.0 seconds Running FinalInlinePass... FinalInlinePass not run because no ProGuard configuration was provided. FinalInlinePass (run) completed in 0.0 seconds Running DelSuperPass... Examined 12359 total methods Found 58 candidate trivial methods Found 55 trivial return invoke-super methods Deleted 55 trivial return invoke-super methods Promoted 0 methods to public visibility Promoted 0 classes to public visibility DelSuperPass (run) completed in 0.0 seconds Running SimpleInlinePass... SimpleInlinePass not run because no ProGuard configuration was provided. SimpleInlinePass (run) completed in 0.0 seconds Running PeepholePass... PeepholePass (run) completed in 0.4 seconds Running ConstantPropagationPass... num_branch_propagated: 0 num_moves_replaced_by_const_loads: 0 ConstantPropagationPass (run) completed in 0.2 seconds Running LocalDcePass... LocalDcePass not run because no ProGuard configuration was provided. LocalDcePass (run) completed in 0.0 seconds Running RemoveUnreachablePass... RemoveUnreachablePass not run because no ProGuard configuration was provided. RemoveUnreachablePass (run) completed in 0.0 seconds Running RemoveGotosPass... Number of unnecessary gotos removed: 187 RemoveGotosPass (run) completed in 0.1 seconds Running DedupBlocksPass... 24 blocks removed DedupBlocksPass (run) completed in 0.1 seconds Running SingleImplPass... Removed interfaces 25 Updated invoke-interface to invoke-virtual 606 SingleImplPass (run) completed in 0.1 seconds Running ReorderInterfacesPass... ReorderInterfacesPass (run) completed in 0.0 seconds Running RemoveEmptyClassesPass... RemoveEmptyClassesPass not run because no ProGuard configuration was provided. RemoveEmptyClassesPass (run) completed in 0.0 seconds Running ShortenSrcStringsPass... src strings shortened 597, 10942 bytes saved ShortenSrcStringsPass (run) completed in 0.0 seconds Running RegAllocPass... Total reiteration count: 635 Total Params spilled early: 59 Total spill count: 4152 Total param spills: 2628 Total range spills: 874 Total global spills: 650 Total splits: 0 Total coalesce count: 11012 Total net moves: -6860 RegAllocPass (run) completed in 2.0 seconds Running CopyPropagationPass... 842 redundant moves eliminated 273 source registers replaced with representative CopyPropagationPass (run) completed in 0.2 seconds Running LocalDcePass... LocalDcePass not run because no ProGuard configuration was provided. LocalDcePass (run) completed in 0.0 seconds Running InterDexPass... InterDexPass not run because no ProGuard configuration was provided. InterDexPass (run) completed in 0.0 seconds Running IRTypeChecker... IRTypeChecker completed in 0.8 seconds Running optimization passes completed in 4.7 seconds Writing out new DexClasses... found 1434 strings from types, 243 from strings in init methods, 7106 total strings Writing optimized dexes completed in 0.7 seconds No method move map data structure! Writing stats completed in 0.1 seconds Freeing global memory completed in 0.0 seconds Done. redex-all main() completed in 5.8 seconds Dex processing finished in 5.88 seconds Repacking dex files Emit Locator Strings: None Creating output apk Creating output APK finished in 0.21 seconds Skipping line number map copy, since no file found to copy Copying line number map v2 map to output dir Copying stats map to output dir Copying src strings map map to output dir Skipping outliner artifacts copy, since no file found to copy Skipping method id map copy, since no file found to copy Skipping class id map copy, since no file found to copy Copying bytecode offset map map to output dir Skipping resources accessed during coldstart copy, since no file found to copy Skipping stats copy, since no file found to copy Skipping resid map after optres pass copy, since no file found to copy Skipping resid map after dedup pass copy, since no file found to copy Skipping resid map after split pass copy, since no file found to copy running merge proguard step redex map is at /tmp/redex.hL1fVg/tmpiu9glax9.redex_dexen/redex_pg_mapping.txt pg map is at None no proguard map file found

[RCoekaerts]

Also how do I attach a debugger when I run redex from the command line?

[justinjhendrick]

--lldb or --gdb depending on your debugger of choice

[RCoekaerts]

Line 705 in ProguardParser.cpp is also not called.

[justinjhendrick]

This is weird and annoying and it shouldn't be so hard to use. Sorry about that.

I think you could just work around this issue by deleting the check inside InterDexPass and rebuilding redex. I'm not even sure why InterDexPass needs the proguard config. I think we put that for internal builds that would surely be wrong if we didn't include our proguard config.

[RCoekaerts]

This solved the issue by the way. It is indeed strange that that check is present in that optimization step, as I would also think it's completely unnecessary. Maybe it was just put there out of habit. For most steps it is probably necessary. Everything seems to work fine now, thanks for your excellent support.

I do have one final, maybe somewhat less important, question. In the app I'm building, I can't observe any speed up. So I tried to create an app in a form I would assume would benefit greatly from this sort of optimisation and compared a build that was optimised with the redex class ordering vs one that was 'optimised' with a random class ordering, but I also didn't have any luck there. Do you maybe have a sample app where the interdex optimisation gains are clearly observable?

[justinjhendrick]

I don't have a ton of experience with InterDex, but I can take a few guesses that could help explain the behavior you're seeing.

• InterDex will have the most impact on large/very large apps. Definitely more than 1 dex, but you may not see any difference in an app with less than 5 dexes (that's a total guess)
• Class ordering matters most on older phones with slow persistent storage and small amounts of ram
• Are you certain InterDex is ordering the classes as you intend? Did you use dexdump or something similar to check?
• Getting a profiler in there could help measure with more precision. See this section. You could also measure disk accesses on the device.
• There are a few more flags you can fiddle with that may help. I'm not sure what they do, other than what the name directly implies:
  • normal_primary_dex controls some extra logic here: https://github.com/facebook/redex/blob/73098a2639dd1946d51d911f26e17c34a19e0ea9/opt/interdex/InterDex.cpp#L492
  • bytecode_sort_mode
  • string_sort_mode
  • here's some example redex config that can control these options (with them set to values we often use)

"InterDexPass" : {
  "normal_primary_dex": false
},
"bytecode_sort_mode": "class_order", 
"string_sort_mode": "class_order"

To verify the sorting options are working you can append CUSTOMSORT:2 to your $TRACE environment variable. Like this: export TRACE=1,IDEX:3,CUSTOMSORT:2

[justinjhendrick]

another bit of advice. I was looking at your redex config that you posted a while ago. Not sure exactly why, but we usually put InterDex pass a bit sooner in the list. Here's where I would suggest putting it:

    "passes": [
      "ReBindRefsPass",
      "BridgePass",
      "SynthPass",
      "FinalInlinePass",
      "DelSuperPass",
      "SimpleInlinePass",
      "PeepholePass",
      "ConstantPropagationPass",
      "LocalDcePass",
      "RemoveUnreachablePass",
      "RemoveGotosPass",
      "DedupBlocksPass",
      "SingleImplPass",
      "ReorderInterfacesPass",
      "RemoveEmptyClassesPass",
      "InterDexPass",
      "ShortenSrcStringsPass",
      "RegAllocPass",
      "CopyPropagationPass",
      "LocalDcePass"
    ]