aws-sdk-java-v2
aws-sdk-java-v2 copied to clipboard
aws
Upgrade apache-httpclient to httpclient5 which uses SLF4J
Describe the feature
I'd like to request an upgrade of the apache httpclient to httpclient5 which uses SLF4J as a logging-facade instead of having to install bridge handlers to redirect log output from commons-logging.
Is your Feature Request related to a problem?
It's not directly a problem, but the issue is that the apache httpclient is leaking a dependency on commons-logging which has to be bridged by either jul-to-slf4j or jcl-over-slf4j which is an inconvenience for downstream libraries.
This feature request is opened in relation to https://github.com/micronaut-projects/micronaut-aws/issues/1203
but it also seems it's a cause of frustration here as well https://github.com/quarkusio/quarkus/issues/4528
Proposed Solution
My proposed solution would be to migrate from httpclient 4.5 til 5.1 following the migration guide set forward at the apache site https://hc.apache.org/httpcomponents-client-5.1.x/migration-guide/index.html
Describe alternatives you've considered
There is a possbility for any downstream library that use dependencies from the aws-sdk-v2 to include bridge handlers to redirect the logs from the httpclient, but imho the solution should be done at the root and trickle down, simplifying library maintainers jobs by not having to actively exclude the commons-logging and installing bridge handlers.
Acknowledge
- [ ] I may be able to implement this feature request
AWS Java SDK version used
2.16.104 and later
JDK version used
11
Operating System and version
Linux
@frehov apologies for the long silence here. Marking this for review from the team.
Hi @frehov thank you for your patience.
Discussed it with the team, and apparently we tried to upgrade before but httpclient5 introduced breaking changes, and the work needed to make the changes on the SDK was not prioritized when compared to other tasks. We currently have no plans to make the upgrade.
Let us know if you have more questions.
It looks like this issue has not been active for more than five days. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please add a comment to prevent automatic closure, or if the issue is already closed please feel free to reopen it.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
@debora-ito could you reopen or re-evaluate this? Last release of Apache http client v4 was released in 2022 and unmaintained since then.
@lazystone I'll raise this for discussion with the team, as it looks like httpcore 4 is at EOL - httpclient 4 will probably follow the same path. Will post an update when we get to a conclusion.
I'd just like to add some weight for the team discussion. httpcore v4 doesn't work with virtual threads and there doesn't seem to be a plan to backport the fix to v4. Therefore the AWS SDK is not compatible with virtual threads until the switch to v5 is made.
To be clear, virtual threads do technically work but they are pinned to their carrier threads due to the use of the synchronized keyword in httpcore. This makes them practically unusable in production.
Is there any plan to support aws sdk2 java with httpclient5 @debora-ito ?
We are considering migrating to httpclient5 and httpcore5, since 4 is at EOL. We need to do research on how to make the change in a backwards compatible way, and how to prioritize this task against other tasks we need to work on. Added a task in our backlog, we don't have a timeline to share right now.
If this ticket is still on your backlog, then I suggest to prioritize #1447 before this one. #1447 solves future compatibility and Virtual Threads usage at the same time.
Until the virtual thread problem is addressed, thoughts on potentially updating documentation (maybe here) to highlight which clients are and are not compatible with virtual threads? Or maybe a higher level "Support Running In Virtual Threads" issue that can track until the SDK is compatible with running in virtual threads?
Background: our team just had a outage due to not knowing the AWS SDK's underlying HTTP client was incompatible with virtual threads. Pre-production stages did not have enough traffic to simulate the errors that result from issues like this until it was unfortunately too late. Given that this issue does not present well until at scale, it might be worth more visibility outside of tracking in this issue and #1447. We had done quick searches of virtual threads and saw this and #1447, but unfortunately assumed they were casual mentions of virtual threads and not that the SDK didn't support virtual threads.
@rhermes62 Sorry this bit you. We're tracking an action item to document our state of support for virtual threads without pinning. We also have a desire to support HTTP clients (e.g. Apache 5.x) that do not have thread pinning issues at some point in the future.
I agree with raising the transparency of client "pitfalls" in documentation and the fact aws uses an unsupported version of the apache client. https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/http-configuration.html would be the perfect spot to mention the usage of apache client 4 vs 5 and expected release date of client5 support.
This issue is open since Nov 9, 2021 , is the there any plan to upgrade to apache client 5.x ?
We have had a lot of deadlocks in the projects I collaborate in because of this issue, until someone found out that the aws sdk was not compatible with vthreads.
Numerous events lost and hours of support added.
Seriously... apache uploaded in July the upgrade, will we have it by 2030? No problem, then Oracle will have done its job...
Hej @debora-ito,
our framework stack for our company provides a builder for all implementations and libraries which uses http client. This is useful for an easy configuration and implementation of customizers. It's bad to hear that aws sdk v2 still uses http client 4 and has no support for http client 5. Therefore, we need to adjust the missing support or duplicate the framework support for http client 4 too.
Maybe aws sdk can also benefit from the openrewrite recipes.
Greetings, Ben
Quick update: yes we are working on the httpclient5 upgrade, it's on early stages of development, no timeline to share.
Thanks for the update, @debora-ito. If upgrading the Apache HttpClient to version 5 is such a big effort, could you share why it is prioritized over implementing a client based on the HTTP client that comes with JDK since Java 11 (see also https://github.com/aws/aws-sdk-java-v2/issues/1447)?
@SoerenHenning The main point is Apache HttpClient is the default sync client in the SDK, we need to upgrade to 5.x since 4.x is deprecated. We've already needed some bugfixes that were released in Apache HttpClient 5.x but were not backported to 4.x.
Just adding our voice. I know @debora-ito you cannot control this, but hopefully data-driven decisisions at amazon include customer counts. Using an EOL (with published unfixed CVEs) client is not acceptable. IMO your choices are retire the AWS client entirely (seems unwise, but business decision) or update it to work with HttpClient 5. And the timeline ideally would have been 2023, but at the very least needs to be by end of year. Thanks again for your own efforts.
