NPE in internal.idn.IdnaMappingTable.findRangesOffset

[ber4444]

trafficstars

       Fatal Exception: java.lang.NullPointerException: Attempt to invoke virtual method 'char java.lang.String.charAt(int)' on a null object reference
       at okhttp3.internal.idn.IdnaMappingTable.findRangesOffset(IdnaMappingTable.java:242)
       at okhttp3.internal.idn.IdnaMappingTable.map(IdnaMappingTable.java:130)
       at okhttp3.internal._HostnamesCommonKt.idnToAscii(-HostnamesCommon.kt:322)
       at okhttp3.internal._HostnamesCommonKt.toCanonicalHost(-HostnamesCommon.kt:307)
       at okhttp3.HttpUrl$Builder.parse$okhttp(HttpUrl.java:1448)
       at okhttp3.HttpUrl$Companion.get(HttpUrl.java:1744)
       at okhttp3.HttpUrl$Companion.parse(HttpUrl.java:1754)
       at okhttp3.HttpUrl.parse(HttpUrl.java:34)
       at zipkin2.reporter.okhttp3.InternalOkHttpSender.newEndpoint(InternalOkHttpSender.java:61)
       at zipkin2.reporter.okhttp3.InternalOkHttpSender.newEndpoint(InternalOkHttpSender.java:36)

seeing the above in some Samsung devices in Crashlytics

we are using '5.0.0-alpha.14'... by the way, would love to see a new alpha, it's been 7 months already

[ber4444]

https://github.com/square/okhttp/blob/02d69bf883c47905d0ad3fd9d2ad75b8ede75bb9/okhttp/src/main/kotlin/okhttp3/internal/idn/IdnaMappingTable.kt#L242 tries to read an index from a String that obviously doesn't exist, easy fix would be to add a sanity test before getting the index

since ranges is String shouldn't this be ranges.codePointAt(entryIndex)?

[yschimke]

I don't think that's the right fix. It's hiding a bug in the logic, the VM/compiler, or the build corrupting the file. We can make this code tolerant of corruption, but it's hiding the root problem.

~For that to be out of range, it must be one of~

1. ~Getting a bad position or limit from sections~
2. ~Some corruption of the data, either on disk or afterwards~
3. ~a logic (okhttp), compilation (agp) or runtime (oem) bug~

Regarding why it's reading a char instead of a codepoint, the encoding is explained at the top of the file. But by this point we are really only looking the compare the last 7 bits of the target.

[yschimke]

I misread the error. It's even stranger than this.

ranges is created here

internal val IDNA_MAPPING_TABLE: IdnaMappingTable = IdnaMappingTable(
      sections =
          "\u0000\u0000
      ranges =
          "\u0000x--AP
      mappings =
          "\u0020̈\u0020
    )

So there isn't something we can fix. This is set and immutable from creation, before any methods are called.

[yschimke]

Can you provide more details about the Samsung devices seeing this error? Are they a particular version? A lot of distinct users? Are they jailbroken?

[swankjesse]

I wonder if packing data into a string is upsetting the runtime somehow.

[ber4444]

It's occasionally happening on Samsung devices only, Android 14. Examples: Galaxy S24 Ultra Galaxy A15 5G

Not rooted

[yschimke]

Anything else common like region?

[Toubap]

Hello Recently happening to me as well.

[yschimke]

If 9% of the errors are from Transsion, I'm more suspect that there is something fishy with those devices.

[piyushgoyal123]

Do we have any fix for this? Happening to me as well.

[yschimke]

Nope, we don't have a repro. So no movement yet.

[yogurtearl]

Using kotlin 2.1.21 and R8 8.10.21 and compile SDK 35

I saw in production on Android 15 using okhttp 5.0.0-alpha.16 and so far only on Pixel devices.

object MyConstants {
         MY_URL = "https://myurl.example.com" 
}

and then we have a call to MyConstants.MY_URL.toHttpUrl()

The code is minified using a recent version of R8 (8.10.21) and works fine 99.99% of the time.

This is a very rare crash... but also surprising in that it is parsing a constant.

Stack trace is a bit different:

          Caused by java.lang.NullPointerException: Attempt to invoke virtual method 'char java.lang.String.charAt(int)' on a null object reference
       at okhttp3.internal.idn.IdnaMappingTable.findRangesOffset(IdnaMappingTable.java:242)
       at okhttp3.internal.idn.IdnaMappingTable.map(IdnaMappingTable.java:130)
       at okhttp3.internal._HostnamesCommonKt.idnToAscii(-HostnamesCommon.kt:322)
       at okhttp3.internal._HostnamesCommonKt.toCanonicalHost(-HostnamesCommon.kt:307)
       at okhttp3.HttpUrl$Builder.parse$okhttp(HttpUrl.kt:1448)
       at okhttp3.HttpUrl$Companion.get(HttpUrl.kt:1744)

As mentioned above, somehow ranges is null on this line, but ranges is set to a constant in the constructor?

val b0 = ranges[entryIndex].code

It happens right on app startup in the first activity.onCreate

[yogurtearl]

Perhaps https://github.com/romainguy/kotlin-explorer could shed light on this?

[michael-winkler]

We also received a crash log.

Fatal Exception: java.lang.RuntimeException: Unable to start activity ComponentInfo{com.mugler.SitemanTicketSystem/com.mugler.sitemanTicketSystem.MainActivity}: java.lang.NullPointerException: Attempt to invoke virtual method 'char java.lang.String.charAt(int)' on a null object reference
       at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:4164)
       at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:4322)
       at android.app.servertransaction.LaunchActivityItem.execute(LaunchActivityItem.java:103)
       at android.app.servertransaction.TransactionExecutor.executeCallbacks(TransactionExecutor.java:139)
       at android.app.servertransaction.TransactionExecutor.execute(TransactionExecutor.java:96)
       at android.app.ActivityThread$H.handleMessage(ActivityThread.java:2685)
       at android.os.Handler.dispatchMessage(Handler.java:106)
       at android.os.Looper.loopOnce(Looper.java:230)
       at android.os.Looper.loop(Looper.java:319)
       at android.app.ActivityThread.main(ActivityThread.java:8919)
       at java.lang.reflect.Method.invoke(Method.java)
       at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:578)
       at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1103)

Caused by java.lang.NullPointerException: Attempt to invoke virtual method 'char java.lang.String.charAt(int)' on a null object reference
       at okhttp3.internal.idn.IdnaMappingTable.findRangesOffset(IdnaMappingTable.java:242)
       at okhttp3.internal.idn.IdnaMappingTable.map(IdnaMappingTable.java:130)
       at okhttp3.internal._HostnamesCommonKt.idnToAscii(-HostnamesCommon.kt:321)
       at okhttp3.internal._HostnamesCommonKt.toCanonicalHost(-HostnamesCommon.kt:306)
       at okhttp3.HttpUrl$Builder.parse$okhttp(HttpUrl.java:1442)
       at okhttp3.HttpUrl$Companion.get(HttpUrl.java:1734)
       at okhttp3.HttpUrl.get(HttpUrl.java:39)
       at retrofit2.Retrofit$Builder.baseUrl(Retrofit.java:554)
       at com.mugler.sitemanTicketSystem.api.Retrofit.retrofitBuilder(Retrofit.kt:48)
       at com.mugler.sitemanTicketSystem.api.Retrofit.init(Retrofit.kt:31)
       at com.mugler.sitemanTicketSystem.api.Retrofit.init$default(Retrofit.kt:24)
       at com.mugler.sitemanTicketSystem.api.ApiServices.getLoginSimulationApi(ApiServices.kt:65)
       at com.mugler.sitemanTicketSystem.utils.ValidationUtil$Companion.sitemanLoginValide(ValidationUtil.java:26)
       at com.mugler.sitemanTicketSystem.MainActivity.validate(MainActivity.kt:729)
       at com.mugler.sitemanTicketSystem.MainActivity.preInitialize(MainActivity.kt:309)
       at com.mugler.sitemanTicketSystem.MainActivity.onCreate(MainActivity.kt:197)
       at android.app.Activity.performCreate(Activity.java:8975)
       at android.app.Activity.performCreate(Activity.java:8944)
       at android.app.Instrumentation.callActivityOnCreate(Instrumentation.java:1456)
       at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:4146)
       at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:4322)
       at android.app.servertransaction.LaunchActivityItem.execute(LaunchActivityItem.java:103)
       at android.app.servertransaction.TransactionExecutor.executeCallbacks(TransactionExecutor.java:139)
       at android.app.servertransaction.TransactionExecutor.execute(TransactionExecutor.java:96)
       at android.app.ActivityThread$H.handleMessage(ActivityThread.java:2685)
       at android.os.Handler.dispatchMessage(Handler.java:106)
       at android.os.Looper.loopOnce(Looper.java:230)
       at android.os.Looper.loop(Looper.java:319)
       at android.app.ActivityThread.main(ActivityThread.java:8919)
       at java.lang.reflect.Method.invoke(Method.java)
       at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:578)
       at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1103)

Samsung Galaxy A54 5G Android 14

And our implementations:

implementation("com.squareup.retrofit2:retrofit:3.0.0")
implementation("com.squareup.retrofit2:converter-gson:3.0.0")
implementation("com.squareup.retrofit2:converter-scalars:3.0.0")
implementation("com.squareup.okhttp3:okhttp:5.1.0")

[gmazzo]

Same here:

It's very strange, since it does not to be fully related to R8. We forced the logic to reach that piece of code in R8 minified code, and it was not failing. But we still have some error reports in production 🤔

[SimonMarquis]

Same here since we upgraded to V5. Though it appears to only happen on Xiaomi devices, in background mode, during the app process initialisation (the stra traces all show Dagger graph initialisation)

[yschimke]

Probably worth filing a bug with the manufacturer then

[michael-winkler]

As mentioned in this GitHub comment:

internal val IDNA_MAPPING_TABLE: IdnaMappingTable = IdnaMappingTable(
    sections = "\u0000\u0000",
    ranges = "\u0000x--AP",
    mappings = "\u0020̈\u0020"
)

Is it possible that R8 or ProGuard is removing this field?. Adding a simple @Keep annotation might help:

@Keep
internal val IDNA_MAPPING_TABLE: IdnaMappingTable = IdnaMappingTable(
    sections = "\u0000\u0000",
    ranges = "\u0000x--AP",
    mappings = "\u0020̈\u0020"
)

@yschimke

[yschimke]

@michael-winkler I mean if it fixes it, sure. But doesn't it need to be non-null to get this far?

aused by java.lang.NullPointerException: Attempt to invoke virtual method 'char java.lang.String.charAt(int)' on a null object reference
       at okhttp3.internal.idn.IdnaMappingTable.findRangesOffset(IdnaMappingTable.java:242)
       at okhttp3.internal.idn.IdnaMappingTable.map(IdnaMappingTable.java:130)

[michael-winkler]

The error indicates that ranges

val b0 = ranges[entryIndex].code

was null at that point in time. This raises the question of how that’s possible, since ranges is defined as a non-null val of type String in the constructor.

internal class IdnaMappingTable internal constructor(
  val sections: String,
  val ranges: String,
  val mappings: String,
)

Could this actually be an R8 or ProGuard issue? Technically, ranges and the other fields are correctly defined, but at runtime they appear to be missing — for whatever reason. This suggests that they might have been stripped or optimized away, despite being used implicitly.

[yschimke]

Could you check the release APK and see what is in there?

[gmazzo]

Could you check the release APK and see what is in there?

IdnaMappingTable.kt

.class public final Lokhttp3/internal/idn/IdnaMappingTable;
.super Ljava/lang/Object;
.source "SourceFile"


# annotations
.annotation runtime Lkotlin/Metadata;
    d1 = {
        "\u0000\n\n\u0002\u0018\u0002\n\u0002\u0010\u0000\n\u0000\u0008\u0000\u0018\u00002\u00020\u0001\u00a8\u0006\u0002"
    }
    d2 = {
        "Lokhttp3/internal/idn/IdnaMappingTable;",
        "",
        "okhttp"
    }
    k = 0x1
    mv = {
        0x2,
        0x2,
        0x0
    }
    xi = 0x30
.end annotation

IdnaMappingTableInstanceKt

.class public final Lokhttp3/internal/idn/IdnaMappingTableInstanceKt;
.super Ljava/lang/Object;
.source "SourceFile"


# annotations
.annotation runtime Lkotlin/Metadata;
    d1 = {
        "\u0000\u0002\n\u0000\u00a8\u0006\u0000"
    }
    d2 = {
        "okhttp"
    }
    k = 0x2
    mv = {
        0x2,
        0x2,
        0x0
    }
    xi = 0x30
.end annotation


# static fields
.field public static final a:Lokhttp3/internal/idn/IdnaMappingTable;


# direct methods
.method static constructor <clinit>()V
    .registers 1

    .line 1
    new-instance v0, Lokhttp3/internal/idn/IdnaMappingTable;

    .line 2
    .line 3
    invoke-direct {v0}, Ljava/lang/Object;-><init>()V

    .line 4
    .line 5
    .line 6
    sput-object v0, Lokhttp3/internal/idn/IdnaMappingTableInstanceKt;->a:Lokhttp3/internal/idn/IdnaMappingTable;

    .line 7
    .line 8
    return-void
.end method

IdnaMappingTableKt

.class public final Lokhttp3/internal/idn/IdnaMappingTableKt;
.super Ljava/lang/Object;
.source "SourceFile"


# annotations
.annotation runtime Lkotlin/Metadata;
    d1 = {
        "\u0000\u0002\n\u0000\u00a8\u0006\u0000"
    }
    d2 = {
        "okhttp"
    }
    k = 0x2
    mv = {
        0x2,
        0x2,
        0x0
    }
    xi = 0x30
.end annotation


# direct methods
.method public static final a(ILjava/lang/String;)I
    .registers 3

    .line 1
    const-string v0, "<this>"

    .line 2
    .line 3
    invoke-static {p1, v0}, Lkotlin/jvm/internal/l;->f(Ljava/lang/Object;Ljava/lang/String;)V

    .line 4
    .line 5
    .line 6
    invoke-virtual {p1, p0}, Ljava/lang/String;->charAt(I)C

    .line 7
    .line 8
    .line 9
    move-result v0

    .line 10
    add-int/lit8 p0, p0, 0x1

    .line 11
    .line 12
    invoke-virtual {p1, p0}, Ljava/lang/String;->charAt(I)C

    .line 13
    .line 14
    .line 15
    move-result p0

    .line 16
    shl-int/lit8 p1, v0, 0x7

    .line 17
    .line 18
    add-int/2addr p1, p0

    .line 19
    return p1
.end method

I'm not a bytecode expect, but it seems all fields of IdnaMappingTable have been removed?

[michael-winkler]

@gmazzo Where have you found that?

That's all I see from the classes.dex

Edit: After I added my mapping I can see more.

.class public abstract Lokhttp3/internal/idn/IdnaMappingTableInstanceKt;
.super Ljava/lang/Object;
.source "SourceFile"


# static fields
.field public static final IDNA_MAPPING_TABLE:Lcom/google/android/material/shape/EdgeTreatment;


# direct methods
.method static constructor <clinit>()V
    .registers 2

    new-instance v0, Lcom/google/android/material/shape/EdgeTreatment;

    const/16 v1, 0x1d

    invoke-direct {v0, v1}, Lcom/google/android/material/shape/EdgeTreatment;->R3.EdgeTreatment.<init>(I)V

    sput-object v0, Lokhttp3/internal/idn/IdnaMappingTableInstanceKt;->IDNA_MAPPING_TABLE:Lcom/google/android/material/shape/EdgeTreatment;

    return-void
.end method

.method public static final a(ILjava/lang/String;)I
    .registers 3

    invoke-virtual {p1, p0}, Ljava/lang/String;->charAt(I)C

    move-result v0

    add-int/lit8 p0, p0, 0x1

    invoke-virtual {p1, p0}, Ljava/lang/String;->charAt(I)C

    move-result p0

    shl-int/lit8 p1, v0, 0x7

    add-int/2addr p1, p0

    return p1
.end method

[gmazzo]

@gmazzo Where have you found that?

It's from Android Studio, by inspecting our production minified release bundle. Looking at the specific classesX.dex

[michael-winkler]

Okay. I did not found it in our release apk but inside our debug apk. I wonder why I can not found it inside the release apk.

Only difference for our build are R8/pro guard is enabled for release only.

[yschimke]

My observation, is that release builds do a lot of mangling, but it's keeping what it needs.

See https://github.com/square/okhttp/pull/8994

I suspect OkHttp is somehow triggering an issue with some different build setup you have. But I don't think the fix is in OkHttp, unless we can make a clean reproduction. I don't want to just sprinkle Keep rules unless we understand why it's being stripped.

I'd suggest specifying the keep rules in your own build for now.

[yschimke]

Probably an unpopular call, but I'm going to close this as I think the PR demonstrates this is working.

I'm very happy to continue discussing, but I don't think it's a bug for OkHttp to fix. I'll re-open in a second, with a repro.

[rolgalan]

Hi @yschimke,

Thanks for providing this test in #8994, with that I've been able to reproduce the issue. The problem actually happens when the proguard-android-optimize.txt file from AGP is added instead of the proguard-android.txt currently used in the test app. The usage of the optimized set of rules is recommended by Android to improve the application performance.

setProguardFiles(listOf(getDefaultProguardFile("proguard-android-optimize.txt"), "proguard-rules.pro"))

To be honest, the test is not failing 🤷 but you can see the IdnaMappingTableInstanceKt actually being stripped out from the final apk. You can validate this by opening the .dex from the apk as explained in above comments. See the screenshots:

normal	optimized

Also, if you add the whyareyoukeeping rule:

-whyareyoukeeping class okhttp3.internal.idn.IdnaMappingTableInstanceKt {
  *;
}

The output with the proguard-android-optimize.txt is that nothing is found to keep it:

Nothing is keeping okhttp3.internal.idn.IdnaMappingTable okhttp3.internal.idn.IdnaMappingTableInstanceKt.getIDNA_MAPPING_TABLE()

Output with the proguard-android.txt:

okhttp3.internal.idn.IdnaMappingTable okhttp3.internal.idn.IdnaMappingTableInstanceKt.getIDNA_MAPPING_TABLE()
|- is invoked from:
|  java.lang.String okhttp3.internal._HostnamesCommonKt.idnToAscii(java.lang.String)
|- is invoked from:
|  java.lang.String okhttp3.internal._HostnamesCommonKt.toCanonicalHost(java.lang.String)
|- is invoked from:
|  okhttp3.HttpUrl$Builder okhttp3.HttpUrl$Builder.parse$okhttp_release(okhttp3.HttpUrl,java.lang.String)
|- is invoked from:
|  okhttp3.HttpUrl okhttp3.HttpUrl$Companion.get(java.lang.String)
|- is invoked from:
|  void okhttp.android.testapp.MainActivity.onCreate(android.os.Bundle)
|- is defined in library method overridden by:
|  okhttp.android.testapp.MainActivity
|- is referenced in keep rule:
|  ./android-test-app/build/intermediates/aapt_proguard_file/release/processReleaseResources/aapt_rules.txt:5:1

With the above information, would you reconsider opening this issue again and potentially adding a keep rule for the IdnaMappingTableInstance in your consumerProguardFile so the host apps don't need to add custom proguard rules to use this library?

[JakeWharton]

I think you should file a bug against R8 because it is doing something incorrect with our code. Keep rules are needed for things accessed reflectively–not for normal references like this.

You can embed a keep rule in your application to work around the problem in the mean time, and then remove it once R8 is fixed (and released). It's unclear when we would be able to remove such a keep rule if we were to add it. After all, we don't know the AGP/R8 versions of our consumers.

[yschimke]

I'll update the test we already have to include those rules. Thanks for finding that.