Migrate the CT log list dependency from v2 to v3

[roger2hk]

Both the v1 and v2 CT log lists will be turned down on 17 October 2022. https://groups.google.com/a/chromium.org/g/ct-policy/c/zejEtWAJtEA/m/fX_AG-HUAQAJ

The latest CT log list URL is https://www.gstatic.com/ct/log_list/v3/log_list.json.

Here is the log_list code search result in this repo. https://github.com/appmattus/certificatetransparency/search?p=1&q=log_list

[rbrauwers]

Anyone knows what will happen after v1 and v2 CT log list shutdown?

The library will stop to work?

[rbrauwers]

I hope the maintainers of the library be 100% transparent about this critical issue. If my understanding is correct, any client of this library will have his http requests refused after Oct 17, 2022, given that the v2 log list will be unaccessible.

More details on this conversation.

cc: @mattmook

[alexandru-lachimov]

I emailed directly to chrome-certificate-transparency team and here is their reply.

[roger2hk]

Joe is right. The painess upgrade is to update the following endpoint from v2 to v3 and the code comments.

https://github.com/appmattus/certificatetransparency/blob/80e835fe1d67d848215a614646ca20e9c3f07026/certificatetransparency/src/main/kotlin/com/appmattus/certificatetransparency/loglist/LogListDataSourceFactory.kt#L49

[alexandru-lachimov]

PR is opened -> https://github.com/appmattus/certificatetransparency/pull/47 FYI @mattmook

[rbrauwers]

Thanks for you support.

Just to be clear, even with a quick release of a version using v3 list log, certificate transparency clients will have a short window to update their apps (˜25 days).

Any help to kindly ask Google to postpone the v2 turndown on 17 October 2022 is appreciated.

[alexandru-lachimov]

Clients had more than one moth to migrate to V3, I have some doubts that guys Google will change their mind and postpone V1 and V2 endpoints shout-down.

[rbrauwers]

My hope is low too. But i think it's worth a try.

Clients has LESS than one month to migrate to V3. This is not much time, specially to libraries that uses certificate transparency, needing to propagate the update to their clients.

[roger2hk]

Can the baseUrl in LogListService be overridden as a workaround before the fix release is ready?

https://github.com/appmattus/certificatetransparency/blob/af503564bf3ee2c2866dafc126d19221254bdd00/certificatetransparency/src/main/kotlin/com/appmattus/certificatetransparency/loglist/LogListDataSourceFactory.kt#L44-L53

[alexandru-lachimov]

Looks like CTTrustManagerBuilder, CTInterceptorBuilder and CTHostnameVerifierBuilder builders have an option to provide custom LogListService using Factory LogListDataSourceFactory.createLogListService("urlToOverride", " okHttpClientToOverride", "timeOutToOverride")

[roger2hk]

Looks like CTTrustManagerBuilder, CTInterceptorBuilder and CTHostnameVerifierBuilder builders have an option to provide custom LogListService using Factory LogListDataSourceFactory.createLogListService("urlToOverride", " okHttpClientToOverride", "timeOutToOverride")

Thanks for looking into that. The log list v3 should be backward compatible with v2, so overriding the baseUrl is the workaround when the fix is not yet released.

[alexandru-lachimov]

Yeap, that looks like to be an option to solve the problem without a release.

[roger2hk]

My hope is low too. But i think it's worth a try.

Clients has LESS than one month to migrate to V3. This is not much time, specially to libraries that uses certificate transparency, needing to propagate the update to their clients.

We'll delay the v2 log list turndown by another month to 2022-11-17. https://groups.google.com/g/certificate-transparency/c/otRk_9FZTEA/m/6UiTRgjsAwAJ

[perwyl]

Looks like CTTrustManagerBuilder, CTInterceptorBuilder and CTHostnameVerifierBuilder builders have an option to provide custom LogListService using Factory LogListDataSourceFactory.createLogListService("urlToOverride", " okHttpClientToOverride", "timeOutToOverride")

Hi @alexandru-lachimov is this sufficient for the workaround? I tested fine but unable to print out the base url to check if it's using v3.

   private fun customCTInterceptorBuilder(): CTInterceptorBuilder {
        val logListV3 = LogListDataSourceFactory.createLogListService(
            baseUrl = CT_LOG_BASE_URL_V3 //"https://www.gstatic.com/ct/log_list/v3/"
        )

        val failureLogger = object : CTLogger {
            override fun log(host: String, result: VerificationResult) {
                Log.d(CT_TAG, "$host -> $result")
                if (result is VerificationResult.Failure) {
                    Tracer.error(CT_TAG, "$host -> $result")
                }
            }
        }

        return CTInterceptorBuilder()
            .setLogListService(logListV3)
            .setLogger(failureLogger)
            .includeHost(XXX")
    }

[alexandru-lachimov]

Hi @perwyl. Yes, it should be fine. You can clone the project and check LogListService usages and its implementation. Hope @mattmook will be back soon to approve and merge this PR. Good news from @roger2hk that turndown V1 and V2 endpoints was delayed.

[roger2hk]

We are pleased to announce that the v2 log list endpoints will serve the v3 log list, which is backward compatible with v2, for another 90 days starting on 2022-11-17. The v2 log list endpoints will start returning 404 on 2023-02-15.

https://groups.google.com/a/chromium.org/g/ct-policy/c/zejEtWAJtEA/m/qOfK8Bk_AgAJ

[kamilargent]

@mattmook could u please merge this PR and release a new version? The endpoints will be shut down next month and we would like to migrate our apps without any workarounds.

[kvetis]

Hi, posting minimal code to fix the interceptor (works for us):

certificateTransparencyInterceptor {
        setLogListService(LogListDataSourceFactory.createLogListService("https://www.gstatic.com/ct/log_list/v3/"))
}

[kamilargent]

Hi, posting minimal code to fix the interceptor (works for us):

certificateTransparencyInterceptor {
        setLogListService(LogListDataSourceFactory.createLogListService("https://www.gstatic.com/ct/log_list/v3/"))
}

I can confirm that it's working - we ended up doing the exact thing a couple of weeks back. I am just worried that not everyone will come across this thread when integrating the library for the 1st time :(

[daniel-tailored]

for the CT provider on trust manager level this should work, minimal code example. let me know if I have overlooked something. thx

installCertificateTransparencyProvider {

    val service = LogListDataSourceFactory.createLogListService(
        baseUrl = "https://www.gstatic.com/ct/log_list/v3/",
        trustManager = delegate
    )

    setLogListService(service)
}

[roger2hk]

Here is the update on v2 log list and CT on Android. https://groups.google.com/g/certificate-transparency/c/38Lr9K46cCA