Segfault with curl backend on macOS

[k3rnL]

Hello,

I'm trying to resolve a bug with sttp on Scala Native on macOS. I have this simple code :

object Main extends App {
  import sttp.client3._
  
  val request = basicRequest
    .get(uri"http://localhost:8123")
    .response(asString)

  println(request.headers)
  val backend = CurlBackend()

  val response = request.send(backend)

  println(response.body)
  println(response.headers)
}

On ubuntu it works as expected. But on macOS I got a segfault in curl_easy_setopt :

* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x27766fdff6e0)
    frame #0: 0x00000001008c4450 libcurl.dylib`Curl_vsetopt + 7988
libcurl.dylib`Curl_vsetopt:
->  0x1008c4450 <+7988>: ldrb   w8, [x8]
    0x1008c4454 <+7992>: cbnz   w8, 0x1008c44a8           ; <+8076>
    0x1008c4458 <+7996>: b      0x1008c445c               ; <+8000>
    0x1008c445c <+8000>: bl     0x10086ffc8               ; Curl_all_content_encodings

So, I tried to understand what's happening, so far I saw that the pointer in the register x8 does not look like a pointer but is a constant value (it never changed across builds and runs) :

(lldb) re r x8
      x8 = 0x000027766fdff6e0

Maybe it is not relevant. I did not figured what is this value. And I'm pretty bad at using lldb, but at least it explain why it segfault.

The backtrace gives us some more information :

(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x27766fdff6e0)
  * frame #0: 0x00000001008c4450 libcurl.dylib`Curl_vsetopt + 7988
    frame #1: 0x00000001008c8bac libcurl.dylib`curl_easy_setopt + 72
    frame #2: 0x00000001001e14c8 test-cpp-scala-native-out`_SM26sttp.client3.curl.CurlApi$D33sttp$client3$curl$CurlApi$$setoptL28scala.scalanative.unsafe.PtrL23scala.Enumeration$ValueL28scala.scalanative.unsafe.PtrL23scala.Enumeration$ValueEO + 96
    frame #3: 0x0000000100142eb4 test-cpp-scala-native-out`_SM39sttp.client3.curl.CurlApi$CurlHandleOpsD6optionL23scala.Enumeration$ValueL16java.lang.StringL29scala.scalanative.unsafe.ZoneL23scala.Enumeration$ValueEO + 96
    frame #4: 0x0000000100210fc8 test-cpp-scala-native-out`_SM32sttp.client3.AbstractCurlBackendD15$anonfun$send$3L28scala.scalanative.unsafe.PtrL29scala.scalanative.unsafe.ZoneL17sttp.model.HeaderL23scala.Enumeration$ValueEPT32sttp.client3.AbstractCurlBackend + 104

Frame 4 learn us that is happens in the send function, digging in the code shows that it is when the headers are passed to curl that the segfault occurs, as it is only one call to curl.option in this method. sttp/core/src/main/scalanative/sttp/client3/AbstractCurlBackend.scala:L54

...
      if (reqHeaders.nonEmpty) {
        reqHeaders.find(_.name == "Accept-Encoding").foreach(h => curl.option(AcceptEncoding, h.value))
        request.body match {
          case _: MultipartBody[_] =>
            headers = transformHeaders(
              reqHeaders :+ Header.contentType(MediaType.MultipartFormData)
            )
          case _ =>
            headers = transformHeaders(reqHeaders)
        }
        curl.option(HttpHeader, headers.ptr) // <--- HERE
      }
...

Today I'm stuck here, do you have any hints or some idea ? I'm available to tests stuff. On my side I'll try to dig more, maybe modify the code of sttp locally, but it looks like a wrong path to me, as it works as expected on other platforms. I start to think that it is a compiler error.

EDIT:

No, my assumption is wrong ! I did not see the line with curl.option(AcceptEncoding, h.value) ! The h.value, is probably my constant value. My only header is this one. I'll dig in this direction. Stay tuned !

[k3rnL]

So, the problems effectively happens in the switch case of CURLOPT_ACCEPT_ENCODING

case CURLOPT_ACCEPT_ENCODING:
    /*
     * String to use at the value of Accept-Encoding header.
     *
     * If the encoding is set to "" we use an Accept-Encoding header that
     * encompasses all the encodings we support.
     * If the encoding is set to NULL we don't send an Accept-Encoding header
     * and ignore an received Content-Encoding header.
     *
     */
    argptr = va_arg(param, char *);
    if(argptr && !*argptr) {
      argptr = Curl_all_content_encodings();
      if(!argptr)
        result = CURLE_OUT_OF_MEMORY;
      else {
        result = Curl_setstropt(&data->set.str[STRING_ENCODING], argptr);
        free(argptr);
      }
    }
    else
      result = Curl_setstropt(&data->set.str[STRING_ENCODING], argptr);
    break;

The assembly code in my first comment, is actually the if(argptr && !*argptr), more exactly the !*argptr. So, in other words, it is the derefencing of the va_arg(...) ptr that segfault. I'm wondering if the whole problem is not a call convention issue. Indeed, in sttp, we pass a Cstring, instead of the expected varargs. But a quick test in my little project does not confirm this theory..

[k3rnL]

All the call to setopt actually segfaults... I commented out the one that was problematic in sttp source code and rebuilt the library, then the next one segfault. And so on. This issue is really bad.

[k3rnL]

Sorry for this answering to myself topic, but I again found some more information. Still in the theory of a call convention problem. Or at least, a compilation problem. I tried to enable optimisation in sbt. And surprise ! It works... at some point. I was able to uncomment all the setopt calls. But now, curl answer with URL_MALFORMAT error. I will become crazy with this issue.

Again, no problem on ubuntu.

java.lang.RuntimeException: Command failed with status URL_MALFORMAT
        at java.lang.StackTrace$.currentStackTrace(Unknown Source)
        at java.lang.Throwable.fillInStackTrace(Unknown Source)
        at sttp.client3.AbstractCurlBackend.lift(Unknown Source)
        at sttp.client3.AbstractCurlBackend.handleBase(Unknown Source)
        at sttp.client3.AbstractCurlBackend.$anonfun$send$1(Unknown Source)

What I did in build.sbt:

nativeConfig ~= {
  _.withLinkingOptions(Seq(
      "/usr/local/lib/libcurl.dylib",
    ))
    .withOptimize(true) // <--- passed this to true
}

[LeeTibbert]

Some data, whilst I am waiting for a PR to pass....

I checked: unit-tests/native/src/test/scala/scala/scalanative/unsafe/CVarArgListTest.scala runs & passes on macOS, so there is hope ;-) (also gets run on Linux, Windows, etc.)

That Test has an interesting bit:

  def vatest(cstr: CString, varargs: Seq[CVarArg], output: String): Unit =
    Zone { implicit z =>
      val buff: Ptr[CChar] = alloc[CChar](1024)
      stdio.vsprintf(buff, cstr, toCVarArgList(varargs))
      val got = fromCString(buff)
      assertTrue(s"$got != $output", got == output)
    }

See the toCVarArgList()? On my next sprint, I need to dive into that to understand what it does on macOS & on Linux etc. Wish me many broken CI runs.

If it were not for Test files, I would never know how to do any of this stuff.

[LeeTibbert]

Later: There is a Variadic functions section in the [Interop Guide] (https://scala-native.readthedocs.io/en/latest/user/interop.html) which may gives some clues as to usage.

toCVarArgList() appears to not be in .scala code, so it is going to take some poking around to find it.

Waiting on one more test, just one tiny one... nativelib/src/main/scala/scala/scalanative/unsafe/CVarArgList.scala is interesting reading. Looks like both X86_64 and Apple silicon (arm64) are supposed to work on macOS.

@k3rnL which hardware are you using (sorry if you said and I missed). If Apple silicon, which JDK are you using? Temurin Java 8 is X86_64 only, so it brings in Rosetta, which might complicate the CVarArgList situation/debugging. Thanks.

[k3rnL]

After morning routine: I read the source code, very interesting indeed !

Also, apple provide a clear explanation on the call convention for arm64. The code in nativelib seems fine to me at first read.

I tried to run the tests you found, but I can't run them for some reason. I didn't dig so much in it, but I tried this snippet in my little test project :

@link("curl")
@extern
object CCurl {
  @name("curl_easy_init")
  def init: Ptr[Curl] = extern

  @name("curl_easy_setopt")
  def setopt(handle: Ptr[Curl], option: CInt, parameter: Ptr[_]): CInt = extern

  @name("curl_easy_setopt")
  def setopt(handle: Ptr[Curl], option: CInt, parameter: CVarArgList): CInt = extern
}

object Main extends App {
  import sttp.client3._

  Zone { implicit z =>
    val curl = CCurl.init
    println("try the stuff")
    CCurl.setopt(curl, 10102, toCString("gzip"))
    println("works!")
  }
}

And... It works 🙃 With or without optimisation.

About Java version :

openjdk version "11.0.15" 2022-04-19 LTS                                                                                                                                          ─╯
OpenJDK Runtime Environment Microsoft-33279 (build 11.0.15+10-LTS)
OpenJDK 64-Bit Server VM Microsoft-33279 (build 11.0.15+10-LTS, mixed mode)
❯ file /opt/jdk-11.0.15+10/Contents/Home/bin/java
/opt/jdk-11.0.15+10/Contents/Home/bin/java: Mach-O 64-bit executable arm64 

So it's all arm64. But ! IntelliJ is running and doing everything with the Temurin. I will change this.

[LeeTibbert]

@k3rnL Thank you for the progress note.

Good to see a sandbox program working. Does that extend to & fix the problem you started out to solve?

I find that gathering information along the way about what hardware, operating systems, & JDKs in use is like picking up bits for a medicine bag. Might be useful someday: knowing what has been seen to work & what has been seen to be flaky.

[adamw]

@LeeTibbert @k3rnL do I understand correctly that this is not an issue in sttp-native code but in scala native itself? Do you know if it got fixed?

[LeeTibbert]

@adamw Thank you for checking if this is current or fixed.
I had thought that that the lines below fixed the problem

 @name("curl_easy_setopt")
  def setopt(handle: Ptr[Curl], option: CInt, parameter: CVarArgList): CInt = extern

The change was to use CVarArgList. @k3rnL will have the final answer.

If it is fixed, the problem was not in Scala Native but that it was not being called as it documents. I've done that before...

Folks, if this is not fixed, please let me know. We want people actively using Scala Native in the field and being happy with it.

[LeeTibbert]

There is, however, one Scala Native MacOS related problem of which folks should be aware. I have a fix in the SN PR queue.

SN 0.4.5 java.lang.Process fails on MacOS 12, Monterrey. Other code works as expected.

The defect is that after years of threatening to do so, Apple deprecated the C vfork() function and made it an alias for fork(). This changed some timing which cascaded and tripped up j.l.Process. This is an OS issue an occurs on both architectures, Intel & Apple (M1).

This defect should have nothing to do with this Issue, but I like to let people know where the edge of the (SN) world is, because this Issue may drive people bug hunting out there.

[k3rnL]

First, in my previous comment I said that it works, this was a wrong statement. It just does not segfault.

@LeeTibbert is correct. The problem comes to the fact that STTP passes a CVarArgList to a "...". It is wrong, because CVarArgList is the equivalent to va_list, the already made up structure from "...". CVarArgList can only be passed to a function that takes a va_list, like vprintf(char *fmt, va_list list) (in contrast with printf(char *fmt, ...)).

Written black on white in the doc

Today, SN does not support the variadic functions as we know them in the end user land. It is really sad, as apparently on x86 machines or non Mac machines, passing a va_list instead of whatever the compiler do when calling a "..." function, works. Otherwise STTP would not work at all...

I found this really interesting Apple doc, just share for someone who may understand it more than me.

So, in CURL lib, there IS a function that takes a va_list, but it is not exported. My quick fix was to wrap the function and let the C compiler do his trick when calling curl_easy_opt :

object CCurl {
....
  @name("my_set_opt") 
  def setopt(handle: Ptr[Curl], option: CInt, parameter: CVarArgList): CInt = extern
....
}

In a C file linked with the project :

void my_set_opt(CURL *curl, CURLoption option, va_list list)
{
   curl_easy_setopt(curl, option, va_arg(list, void*));
}

And this works like a charm in my sandbox program. And should work as well in STTP. The best would be to use : in setopt.h

CURLcode Curl_vsetopt(struct Curl_easy *data, CURLoption option, va_list arg);

But as I said, it seems that it is not exported, but if you are luckier than me and found how to use it directly, do not worry about curl_easy_setopt, it's only here to check the CURL pointer and create the va_list... :

CURLcode curl_easy_setopt(struct Curl_easy *data, CURLoption tag, ...)
{
  va_list arg;
  CURLcode result;

  if(!data)
    return CURLE_BAD_FUNCTION_ARGUMENT;

  va_start(arg, tag);

  result = Curl_vsetopt(data, tag, arg);

  va_end(arg);
  return result;
}

[Pask423]

If I understand you correctly then unitil CURL team decides to export this particular function there is no possibility to fix this problem in STTP ?

[k3rnL]

It is possible through a wrapper to pass the parameter to as a pointer from SN to C, convert the pointer in a var arg inside the wrapper and then to CURL.

I can make an MR if you wish

[Pask423]

Great idea, thanks

[LeeTibbert]

Since this is likely to be a general concern for a small community, would it be possible to post a URL for the fix. If agreeable, I would like to study it and/or pass the URL along.

This sounds like an issue/mole that may well get solved here but which will be popping up to visit me in other guises. Thank you.

Good to see that solutions are being found.