curb should expose curl_easy_setopt

[kagminjeong]

As detailed in issue #48, curb does not directly allow setting certain timeout options available in curl. This would have been a minor inconvenience if curb exposed curl_easy_setopt call (and corresponding multi-call, if any). Even without any CURLOPT_* constants defined I could still pass appropriate magic numbers to curl_easy_setopt and have a workaround for timeout functionality until a better solution is put in place. Currently I have no choice but to edit C extension code, followed by distributing and installing custom packages to/on a number of development and production systems.

Curb should expose curl_easy_setopt, perhaps with a warning that it is possible to break things by passing certain values to it (e.g. options that take C strings as arguments for example), for cases when it does not provide a "native" interface to some functionality.

[taf2]

or you send me a patch and if looks solid has some test coverage i'd add it to the next release...

[kagminjeong]

I thought you might say that :)

I filed the ticket first in case the functionality already existed and I just couldn't find it.

[ghazel]

+1

I had a need for this as well ("delete=" and "head=" are implemented, but not "delete" and "head" to return a boolean).

[taf2]

yep, libcurl bindings for ruby are in need of a rewrite... i just don't have the time.

[kagminjeong]

I started working on this and testing of setopt passthrough is rather tricky.

By design curb issues setopt calls when curb#perform is called, and forwarded setopts happen before that. This makes it impossible to use setopt passthrough to set an option that is explicitly supported by curb.

This, in turn, means that to test that setopt passthrough works we need to use options that are not supported by curb. This is not only a relatively small and obscure set, but one that is shrinking with time as curb gains features. It may be the case that for less frequently used value types there simply are not any options that curb does not natively handle, and so such types would be untestable.

This could be avoided by storing the options and values given to curb's setopt in a list and issuing them to libcurl after curb sets its own options. This should work reasonably well but obviously requires memory (including dynamic allocation) to store the options.

[taf2]

yeah my real thought here is i agree and I think the design of the original gem needs a rewrite... IMO it would better serve us to have libcurl bindings that implemented a thin passthrough over the libcurl easy/multi/shared interfaces.

Ideally, this would include the socket interfaces allowing us to use eventmachine or cool.io (maybe) as event loops... it also means a lot less C and a lot more ruby.

I have a local branch I need to push - I've had it for too long just sitting. have wanted to get this written for almost 2 years now...

[taf2]

Keeping this ticket open. There is a set interface now, but it's still very minimal as far as what can be set.

[apollo-io]

Hi - firstly, what a great package curb is, and language ruby is. I'm just starting out with learning ruby and using it to write some Rspec test cases for my team to use.

I'm currently having a related issue trying to setopt for Curl::CURLOPT_RESOLVE, receiving the message that setting that option is not supported.

I'm looking to replicate the following command:

curl --resolve "mytest.com:443:127.0.0.1" https://mytest.com

I have a working test that uses ssl_verify_host = false, which is working for most of my test cases, but is not working for end to end tests that need the hostname to be set.

Why on earth would you want to do that? The load balancers I'm connecting to perform logic based on the url used for the connection (SNI?) and therefore this drops my connection request.

I'm looking at invoking the command line option to do this testing at the moment, which would be a real shame.

Thank you very much for any help you can provide.

[driehuis]

Implementing a setter interface to CURLOPT_RESOLVE would be rather a challenge, as it involves setting up and maintaining a list of structs just to hold the value).

As an aside, I can confirm @apollo-io 's issue: lack of SNI support is causing hard to debug issues with programs written to use Curb (once you dig out the underlying OpenSSL error, the user is likely to go on a wild goose chase for TLSv1 vs SSLv3 mismatches).

The problem exhibits itself when connecting to an SNI enabled server on a different IP than that given in DNS. This requires the Curb user to mess around with tricks like giving the URL as https://127.0.0.1:443/ whilst setting the requested server in the headers["Host"] parameter. In that case, OpenSSL doesn't know it should pass the Host header to the HELLO negociation, and SNI fails with a rather confusing error message 14077410 from OpenSSL.

The easy.set interface may not be the optimal way to implement this. Perhaps an easier way is to extend the URL syntax to allow specifying https://www.example.com:443:127.0.0.1/, rewriting that to https://www.example.com/ before passing it to Curl and set the CURLOPT_RESOLVE structure as required

Yet another alternative is to add an easy.connect_ip attribute that optionally overrides the DNS lookup (again, probably using CURLOPT_RESOLVE under the hood).

[taf2]

@driehuis are you saying curb doesn't support SNI? This seems like something, that should be supported by default... e.g.

res = Curl.get("https://www.google.com/")
puts res.header_str
puts res.body_str

This should automatically be doing an SNI lookup?

[edit] - this isn't a curb issue but rather a lack of SNI support by the system installed OpenSSL?

[driehuis]

@taf2: Curb does support SNI, but only if you wish to connect to the IP address as resolved through DNS.

If you need to check stuff behind a load balancer, or if for other reasons you need to avoid the DNS lookup, exposing the functionality of CURLOPT_RESOLVE is required to make SNI work. The trickery that I described above only works for plain HTTP and fails miserably for SSL (as my users deftly pointed out, rubbing in that the PHP bindings to curl do expose this feature. I told 'm where to stuff it, but I digress).

My use case is checking if multiple backends for the same servername are all in sync, so IP address is the only discriminator.