In what cases are updates by TSSslSecretSet() actually used?

[ywkaras]

I'm running a test with this ssl_multicert.config file:

dest_ip=* ssl_cert_name=2050.crt ssl_key_name=private.key

The cert in 2050.crt expires in the year 2050. The steps of the test are:

1. Execute a curl that makes a TLS connection, but does not authenticate the received server cert, with verbose output enabled.
2. Use TSSslSecretSet()/TSSslSecretUpdate() to change the 2050.crt to one that is identical, except for expiring in 2060.
3. Repeat the same curl command.

Here is the debug trace output with the debug regex ssl: https://gist.github.com/ywkaras/a7b511a290d62627be18ba301653944c The update of the cert can be seen starting a line 201. However, the expiration date in the verbose output in the second curl does not change. Presumably this means the new cert is not used.

I notice that, in this function: https://github.com/apache/trafficserver/blob/de8e2439fdd4dc491bb7af512b8f8603a81a7be1/iocore/net/SSLNetVConnection.cc#L1129 getCTX() is only called in the client case, not the server case. Does this limit the cases where updates to secrets are actually used?

[ywkaras]

@shinrich @maskit any input on this?

[shinrich]

It should work in the server side as well as I recall. It definitely should work in the client side case. Yahoo has a plugin that uses these calls. You could review which calls are being used there.

[ywkaras]

It should work in the server side as well as I recall. It definitely should work in the client side case. Yahoo has a plugin that uses these calls. You could review which calls are being used there.

From the debug traces it looks like all correct calls are being made, but ATS nonetheless seems to send the un-updated server cert to curl.

[maskit]

I wonder if the plugin that calls TSSslSecretSet is something you use on prod or just a test plugin. Is the API called at a right timing (hook)?

[ywkaras]

I wonder if the plugin that calls TSSslSecretSet is something you use on prod or just a test plugin. Is the API called at a right timing (hook)?

Yes it's a prod plugin. TSSslSecretSet() is called from a periodic scheduled event handler, not from a hook.

[maskit]

You mean schedule_every()? I'm not familiar with how this API works, but the documentation suggests you use it at TS_LIFECYCLE_SSL_SECRET_HOOK, although it may not necessarily mean you can't use it at different timings.

[ywkaras]

@maskit do you have a test plugin for these APIs you could open source?

[maskit]

No, I have never used the API.

[shinrich]

@ywkaras could you make a test plugin that exercises the paths used by the Yahoo plugin? Then folks outside of Yahoo could help you debug. And we could make sure that the secret API doesn't regress.

[ywkaras]

@ywkaras could you make a test plugin that exercises the paths used by the Yahoo plugin? Then folks outside of Yahoo could help you debug. And we could make sure that the secret API doesn't regress.

OK I'll write a simple plugin as a part of a new Au test.

[ywkaras]

The Au test in this PR, https://github.com/apache/trafficserver/pull/9587 , illustrates a case where a update to an X509 cert does not seem to get used.

[ywkaras]

I cherry-picked the new test onto what I think is the first commit with the TSSslSecretXxx API functions: https://github.com/ywkaras/trafficserver/tree/test_ts_ssl_old . The result was the same. Here is the trace output from the test on this branch: https://gist.github.com/ywkaras/88bb0c7e8bd39072bbc0d877a8714121 .

[maskit]

Maybe I need to read the test closely, but I'm confused. You said the API doesn't works as expected but the autest passes.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. Marking it stale to flag it for further consideration by the community.

[thebadpete]

Hi, this is Peter from Yahoo, we are seeing the issues in our prod hosts. It used to work some time back last year, but now we are seeing the issues everything there are cert/key updates. The cert/key fetched from secure storages are not taking effect in memory.