kubo Missing support fr RPC API over HTTPS

Checklist

[X] This is a bug report, not a question. Ask questions on discuss.ipfs.tech.
[X] I have searched on the issue tracker for my bug.
[X] I am running the latest kubo version or have an issue updating.

Installation method

ipfs-update or dist.ipfs.tech

Version

/ipfs version --all
Kubo version: 0.30.0
Repo version: 16
System version: amd64/linux
Golang version: go1.22.7

Config

(probably not relevant)

{"API":{"HTTPHeaders":{}},"Addresses":{"API":"/ip4/127.0.0.1/tcp/5001","Announce":[],"AppendAnnounce":[],"Gateway":"/ip4/127.0.0.1/tcp/8080","NoAnnounce":[],"Swarm":["/ip4/0.0.0.0/tcp/4001","/ip6/::/tcp/4001","/ip4/0.0.0.0/udp/4001/quic-v1","/ip4/0.0.0.0/udp/4001/quic-v1/webtransport","/ip6/::/udp/4001/quic-v1","/ip6/::/udp/4001/quic-v1/webtransport"]},"AutoNAT":{},"Bootstrap":["/dnsaddr/bootstrap.libp2p.io/p2p/QmNnooDu7bfjPFoTZYxMNLWUQJyrVwtbZg5gBMjTezGAJN","/dnsaddr/bootstrap.libp2p.io/p2p/QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa","/dnsaddr/bootstrap.libp2p.io/p2p/QmbLHAnMoJPWSCR5Zhtx6BHJX9KiKNN6tpvbUcqanj75Nb","/dnsaddr/bootstrap.libp2p.io/p2p/QmcZf59bWwK5XFi76CZX8cbJ4BhTzzA3gU1ZjYZcYW3dwt","/ip4/104.131.131.82/tcp/4001/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ","/ip4/104.131.131.82/udp/4001/quic-v1/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ"],"DNS":{"Resolvers":{}},"Datastore":{"BloomFilterSize":0,"GCPeriod":"1h","HashOnRead":false,"Spec":{"mounts":[{"child":{"path":"blocks","shardFunc":"/repo/flatfs/shard/v1/next-to-last/2","sync":true,"type":"flatfs"},"mountpoint":"/blocks","prefix":"flatfs.datastore","type":"measure"},{"child":{"compression":"none","path":"datastore","type":"levelds"},"mountpoint":"/","prefix":"leveldb.datastore","type":"measure"}],"type":"mount"},"StorageGCWatermark":90,"StorageMax":"10GB"},"Discovery":{"MDNS":{"Enabled":true}},"Experimental":{"FilestoreEnabled":false,"Libp2pStreamMounting":false,"OptimisticProvide":false,"OptimisticProvideJobsPoolSize":0,"P2pHttpProxy":false,"StrategicProviding":false,"UrlstoreEnabled":false},"Gateway":{"DeserializedResponses":null,"DisableHTMLErrors":null,"ExposeRoutingAPI":null,"HTTPHeaders":{},"NoDNSLink":false,"NoFetch":false,"PublicGateways":null,"RootRedirect":""},"Identity":{"PeerID":"12D3KooWHmUkKz2ys3uQpNStsCwDP7DBpvPMMF4182KsBQ55xb1A"},"Internal":{},"Ipns":{"RecordLifetime":"","RepublishPeriod":"","ResolveCacheSize":128},"Migration":{"DownloadSources":[],"Keep":""},"Mounts":{"FuseAllowOther":false,"IPFS":"/ipfs","IPNS":"/ipns"},"Peering":{"Peers":null},"Pinning":{"RemoteServices":{}},"Plugins":{"Plugins":null},"Provider":{"Strategy":""},"Pubsub":{"DisableSigning":false,"Router":""},"Reprovider":{},"Routing":{"Methods":null,"Routers":null},"Swarm":{"AddrFilters":null,"ConnMgr":{},"DisableBandwidthMetrics":false,"DisableNatPortMap":false,"RelayClient":{},"RelayService":{},"ResourceMgr":{},"Transports":{"Multiplexers":{},"Network":{},"Security":{}}}}

Description

When trying to connect to a distant API over https, it fails and try to connect with http.

/ipfs --api /dns/myapihost.mydomain.com/tcp/443/https files stat /
2024-10-07T16:00:52.505+0200	ERROR	cmds/http	http/parse.go:212	could not guess encoding from content type "text/html"
Error: unknown error content type: text/html

When checking logs from the RP we have in place receiving the request, we see "scheme": "http".

When trying to access directly the same URL with curl and https, it works properly.

Oct 07 '24 14:10 ze42

Thank you for reporting this bug.

Indeed, /https and /tls/http are not supported. https:// hint is not forwarded to go-ipfs-cmds client.

The host value in:

https://github.com/ipfs/kubo/blob/091bc083c3cb76ff408a9df2c020b0cbc49f141b/cmd/ipfs/kubo/start.go#L357

likely does not include protocol, just host, and http:// is added in:

https://github.com/ipfs/go-ipfs-cmds/blob/ae8443f905e4604e2a0c001ac20a7c8fe4dd7948/http/client.go#L87-L89

Potential quick fix is to make sure kubo/cmd/ipfs/kubo/start.go prepends https:// to host if apiAddr contains /https or /tls/http

@ze42 would you have time to open PR with a fix?

Oct 08 '24 22:10 lidel

@lidel I’m new to this repository, but I believe this fix requires a small change, and I would like to contribute to it. Can I take this?

Oct 22 '24 12:10 Sairaviteja27

@Sairaviteja27 sure, submit PR once you have something ready for review 👍

Oct 23 '24 19:10 lidel

Hi,

Just wanted to share an update. I was occupied with some personal work and initially faced a few challenges due to the earlier version. However, I updated to the latest version (0.32.1) and tried to reproduce the issue, but I didn't encounter any errors.

Next, I will try downgrading my IPFS version to check if the issue persists and also investigate further by cloning the repository.

Dec 01 '24 01:12 Sairaviteja27

After some investigation, I found that if I prepend https:// when the apiAddr contains /https or /tls/http, the cmdhttp.NewClient method call prepends http:// again to the address. I tried changing the serverAddress, but it’s not possible because both the serverAddress and client are unexported. The only solution I can think of is adding a function like this in the go-ipfs-cmds library and using it as an option in start.go:

func ClientWithHTTPSServerAddress() ClientOpt {
    return func(c *client) {
        c.serverAddress = strings.Replace(c.serverAddress, "http://", "https://", 1)
    }
}

This would allow the client to send HTTPS requests, but additional changes may be needed for handling HTTPS responses. When I tried this, I encountered the error: http: server gave HTTP response to HTTPS client

@lidel @gammazero Correct me if I’m mistaken or overlooked anything, and please advise on the next steps for resolving this issue.

Dec 29 '24 14:12 Sairaviteja27

However, I updated to the latest version (0.32.1) and tried to reproduce the issue, but I didn't encounter any errors.

Sorry, but does it work or not in the latest version?

Jan 03 '25 12:01 hsanjuan

However, I updated to the latest version (0.32.1) and tried to reproduce the issue, but I didn't encounter any errors.

Sorry, but does it work or not in the latest version?

Hi, I initially connected with latest version using 5001 and random ports such as 8182 without encountering any errors, but the RPC calls worked only over HTTP, not HTTPS.

The issue became apparent when I attempted to use standard ports (443, 80) on the remote node. Since these ports were already in use, I couldn’t run IPFS on these ports there.

After replicating the environment with a version downgrade, I observed the same behavior.

It seems the OP's concern was making RPC calls over HTTPS, which might require changes as mentioned in my previous comment. https://github.com/ipfs/kubo/issues/10539#issuecomment-2564739293

Jan 03 '25 13:01 Sairaviteja27