Add support for network interfaces in requests

This change modifies the metadata retrieval logic to handle IPv6 link-local addresses that require a specific network interface (scope_id). Previously, cloud-init failed to connect when using addresses like [fe80::a9fe:a9fe%25{iface}] due to improper handling in the underlying requests layer.

By introducing a scoped connection handling approach, this fix enables cloud-init to operate correctly in IPv6-only OpenStack environments and similar setups where metadata services are accessible via scoped link-local addresses.

Summary of Patch Fixes:

1. Interface in URL: Added proper support for link-local IPv6 addresses with network interface identifiers (e.g., %eth0), enabling reliable socket binding using SO_BINDTODEVICE.

2. Enhanced URL Parsing: Improved URL parser to correctly handle hostnames, ports (with defaults), interfaces, and support for both % and %25 encoded formats across various schemes including http, https, ftp, sftp, and file.

Fixes #6205

How did you test? What was the test result? Please include evidence that this works. Bug fixes should include test coverage, please see this and this for getting started.

I just noticed the dependency. New dependencies are strongly discouraged. Is this dependency already available in Linux distributions? Was an alternative approach considered? Why isn't it possible to do the same thing without the new dependency?

Not commenting on the code itself, but I think this approach is the right one until such a time as requests handles scoped link-local addresses correctly. Given the general tone of https://github.com/psf/requests/pull/6927 I wouldn't count on that happening any time soon.

The additional dependency is somewhat problematic, though. Requests-toolbelt is available in Debian and Fedora, but doesn't seem to be included in Fedora-derivatives (checked Rocky 8 and 9, anyway).

@nmeyerhans Please check again Logs : cloud-init.log cloud-init-output.log

@xiachen-rh Can we do some testing on this change on RHEL-10 and using the same repro case we had before? Also does this fix your test case https://github.com/psf/requests/issues/6735#issuecomment-2560736811 ?

@ani-sinha OK, let me test it

Hello! Thank you for this proposed change to cloud-init. This pull request is now marked as stale as it has not seen any activity in 14 days. If no activity occurs within the next 7 days, this pull request will automatically close.

If you are waiting for code review and you are seeing this message, apologies! Please reply, tagging TheRealFalcon, and he will ensure that someone takes a look soon.

(If the pull request is closed and you would like to continue working on it, please do tag TheRealFalcon to reopen it.)

@ani-sinha I tested this changes on OpenStack env with ipv6 only network, cloud-init can get metadata via ipv6, so cloud-init can work without changing the code of 'requests'.

@ani-sinha I tested this changes on OpenStack env with ipv6 only network, cloud-init can get metadata via ipv6, so cloud-init can work without changing the code of 'requests'.

@holmanb what are we going to do with this PR? I have not followed the fix closely myself so can't comment. Do you think we can merge it to cloud-init?

@ani-sinha I tested this changes on OpenStack env with ipv6 only network, cloud-init can get metadata via ipv6, so cloud-init can work without changing the code of 'requests'.

@holmanb what are we going to do with this PR? I have not followed the fix closely myself so can't comment. Do you think we can merge it to cloud-init?

Someone has to review URLParserPlus which is the crux of the change. Does it handle all URL types? Also run integration tests to make sure there are no functional breakage.

Hello! Thank you for this proposed change to cloud-init. This pull request is now marked as stale as it has not seen any activity in 14 days. If no activity occurs within the next 7 days, this pull request will automatically close.

If you are waiting for code review and you are seeing this message, apologies! Please reply, tagging TheRealFalcon, and he will ensure that someone takes a look soon.

(If the pull request is closed and you would like to continue working on it, please do tag TheRealFalcon to reopen it.)

@rezaf28 , also, before we can accept your change, you need to sign the CLA. Please note that Canonical's CLA process has changed recently, so if you signed a long time ago, you are required to resign in order to contribute.

Hello! Thank you for this proposed change to cloud-init. This pull request is now marked as stale as it has not seen any activity in 14 days. If no activity occurs within the next 7 days, this pull request will automatically close.

If you are waiting for code review and you are seeing this message, apologies! Please reply, tagging TheRealFalcon, and he will ensure that someone takes a look soon.

(If the pull request is closed and you would like to continue working on it, please do tag TheRealFalcon to reopen it.)