Hint can't find symbol

[polymorphicengine]

Hello, some days ago tidal was updated to have link support, this is done by the following package https://hackage.haskell.org/package/tidal-link.

unfortunately this seems to break running tidal inside a hint interpreter. Here is a minimal example:

module Main where

import Language.Haskell.Interpreter

main :: IO ()
main = do
  x <- runInterpreter interpreter
  case x of
    Left e -> putStrLn (show e)
    Right _ -> return ()

interpreter :: Interpreter ()
interpreter = do
         setImportsF [ModuleImport "Sound.Tidal.Context" NotQualified NoImportList, ModuleImport "Prelude" NotQualified NoImportList]
         runStmt "tidal <- startTidal (superdirtTarget {oLatency = 0.1, oAddress = \"127.0.0.1\", oPort = 57120}) (defaultConfig {cVerbose = True, cFrameTimespan = 1/20})"

it throws the following errors:

hint-try: /home/martin/.cabal/store/ghc-9.2.3/tidal-link-1.0.1-e904e203bf8f1a5046be19d3e454931fcff763f9b9920cad70d59e090bc3569f/lib/libHStidal-link-1.0.1-e904e203bf8f1a5046be19d3e454931fcff763f9b9920cad70d59e090bc3569f.a: ELF TLSGD relocation for symbol `_ZN4asio6detail15keyword_tss_ptrINS0_10call_stackINS0_14thread_contextENS0_16thread_info_baseEE7contextEE6value_E' not supported on the target platform
hint-try: /home/martin/.cabal/store/ghc-9.2.3/tidal-1.9.2-54c2e1be48d3c91b38aa9169eaa072a94363b15bbbc85dac3185686222ca970e/lib/libHStidal-1.9.2-54c2e1be48d3c91b38aa9169eaa072a94363b15bbbc85dac3185686222ca970e.a: unknown symbol `abl_link_create_session_state'
hint-try: ^^ Could not load 'tidalzm1zi9zi2zm54c2e1be48d3c91b38aa9169eaa072a94363b15bbbc85dac3185686222ca970e_SoundziTidalziStream_Target_con_info', dependency unresolved. See top entry above.

hint-try: ^^ Could not load 'tidalzm1zi9zi2zm54c2e1be48d3c91b38aa9169eaa072a94363b15bbbc85dac3185686222ca970e_SoundziTidalziStream_Target_static_info', dependency unresolved. See top entry above.

GhcException "\nGHC.ByteCode.Linker.lookupIE\nDuring interactive linking, GHCi couldn't find the following symbol:\n  tidalzm1zi9zi2zm54c2e1be48d3c91b38aa9169eaa072a94363b15bbbc85dac3185686222ca970e_SoundziTidalziStream_Target_con_info or tidalzm1zi9zi2zm54c2e1be48d3c91b38aa9169eaa072a94363b15bbbc85dac3185686222ca970e_SoundziTidalziStream_Target_static_info\nThis may be due to you not asking GHCi to load extra object files,\narchives or DLLs needed by your current session.  Restart GHCi, specifying\nthe missing library using the -L/path/to/object/dir and -lmissinglibname\nflags, or simply by naming the relevant files on the GHCi command line.\nAlternatively, this link failure might indicate a bug in GHCi.\nIf you suspect the latter, please report this as a GHC bug:\n  https://www.haskell.org/ghc/reportabug\n"

note that running the same line of code in a ghci session works fine. i'm really lost here, so i hope someone can help!

[gelisam]

I need more information.

1. What does "link support" mean, is this music lingo or do you mean linking in the sense of ld and .so/.dll files?
2. Would it be difficult to further minimize the example by importing a minimal module rather than all of Tidal?
3. Am I correct that those hint-try: ... errors are from cabal, not hint, and thus the problem occurs before hint can do anything about it?

[gelisam]

note that running the same line of code in a ghci session works fine.

Do you start your ghci session in a way which also causes those hint-try: ... errors to appear?

[polymorphicengine]

thanks for the quick reply,

1. Here 'link' is indeed in reference to a music protocol and has nothing to do with linking in the sense of ld
2. I can give it a try, maybe by just importing tidal-link itself
3. Yes, those errors seem to come from cabal (but only occur once running the interpreter), hint throws the last GHCExpception error

Do you start your ghci session in a way which also causes those hint-try: ... errors to appear?

no, in ghci i just import Sound.Tidal.Context and run

tidal <- startTidal (superdirtTarget {oLatency = 0.1, oAddress = "127.0.0.1", oPort = 57120}) (defaultConfig {cVerbose = True, cFrameTimespan = 1/20})

which throws no errors and starts the tidal stream as expected

[polymorphicengine]

the following only imports tidal-link and fails with basically the same error:

module Main where

import Language.Haskell.Interpreter

main :: IO ()
main = do
  x <- runInterpreter interpreter
  case x of
    Left e -> putStrLn (show e)
    Right _ -> return ()

interpreter :: Interpreter ()
interpreter = do
         setImportsF [ModuleImport "Sound.Tidal.Link" NotQualified NoImportList, ModuleImport "Prelude" NotQualified NoImportList]
         runStmt "s <- createSessionState"

only importing tidal-link, without executing any code related to it works fine.

i tried some other functions from https://hackage.haskell.org/package/tidal-link-1.0.1/docs/Sound-Tidal-Link.html aswell and all seem to fail..

zalastax, author of tidal-link suggested to call ghc with argument "-Lstdc++", so i tried using unsafeRunInterpreterWithArgs ["-Lstdc++"], but it results in the same error message too

[gelisam]

Let's take a closer look at the error messages.

The hint-try: ... messages complain about the following symbols:

1. _ZN4asio6detail15keyword_tss_ptrINS0_10call_stackINS0_14thread_contextENS0_16thread_info_baseEE7contextEE6value_E
2. abl_link_create_session_state
3. tidalzm1zi9zi2zm54c2e1be48d3c91b38aa9169eaa072a94363b15bbbc85dac3185686222ca970e_SoundziTidalziStream_Target_con_info
4. tidalzm1zi9zi2zm54c2e1be48d3c91b38aa9169eaa072a94363b15bbbc85dac3185686222ca970e_SoundziTidalziStream_Target_static_info

The shape of these symbols tell us that they are, respectively:

1. a C++ mangled name
2. a C function name
3. a ghc z-encoded name
4. a ghc z-encoded name

And once we decode those names, we get something a lot more readable:

1. asio::detail::keyword_tss_ptr<asio::detail::call_stack<asio::detail::thread_contextasio::detail::thread_info_base>::context>::value_
2. abl_link_create_session_state 3.tidal-1.9.2-012aac824dcb46a140aefc0b0d9e9c6e5504cdb417cbaf42de9ebe02634045b9_Sound.Tidal.Stream_Target_con_info
3. tidal-1.9.2-012aac824dcb46a140aefc0b0d9e9c6e5504cdb417cbaf42de9ebe02634045b9_Sound.Tidal.Stream_Target_static_info

[gelisam]

3. Yes, those errors seem to come from cabal (but only occur once running the interpreter), hint throws the last GHCExpception error

By adding a print statement before the runInterpreter call, or simply by compiling the program with cabal and then running the executable manually afterwards, I have determined that on the contrary, the errors are thrown by the executable, not by cabal, somewhere during the execution of the startTidal line.

[gelisam]

The error occurs even if the interpreted code is simplified to

runStmt "let x = superdirtTarget"

but not if it is further simplified to

runStmt "let x = defaultConfig"

Therefore, the issue doesn't happen when the startTidal or createSessionState IO action is executed, but rather, when the module which provides superdirtTarget is loaded. Note that it is lazily loaded, which is why importing that module is insufficient to trigger the error.

[gelisam]

I see you've added a tidal-link-1.0.1 exporting an extra hello :: IO () action. Thanks, this helps to eliminate a whole bunch of code, as the following still fails:

module Main where
import Language.Haskell.Interpreter

main :: IO ()
main = do
  x <- runInterpreter interpreter
  case x of
    Left e -> do
      -- GhcException "\nByteCodeLink.lookupCE\nDuring interactive linking, GHCi couldn't find the following symbol:\n  tidalzmlinkzm1zi0zi1zm0358d4e416ad055e1e148187987824589541f00f08ddb5855faa8ff4af619c66_SoundziTidalziLink_hello_closure\nThis may be due to you not asking GHCi to load extra object files,\narchives or DLLs needed by your current session.  Restart GHCi, specifying\nthe missing library using the -L/path/to/object/dir and -lmissinglibname\nflags, or simply by naming the relevant files on the GHCi command line.\nAlternatively, this link failure might indicate a bug in GHCi.\nIf you suspect the latter, please report this as a GHC bug:\n  https://www.haskell.org/ghc/reportabug\n"
      putStrLn (show e)
    Right _ -> do
      return ()

interpreter :: Interpreter ()
interpreter = do
  setImports ["Prelude", "Sound.Tidal.Link"]
  -- toy: /home/gelisam/.cabal/store/ghc-8.10.7/tidal-link-1.0.1-0358d4e416ad055e1e148187987824589541f00f08ddb5855faa8ff4af619c66/lib/libHStidal-link-1.0.1-0358d4e416ad055e1e148187987824589541f00f08ddb5855faa8ff4af619c66.a: unknown symbol `_ZN4asio6detail15keyword_tss_ptrINS0_10call_stackINS0_14thread_contextENS0_16thread_info_baseEE7contextEE6value_E'
  -- toy: /home/gelisam/.cabal/store/ghc-8.10.7/tidal-link-1.0.1-0358d4e416ad055e1e148187987824589541f00f08ddb5855faa8ff4af619c66/lib/libHStidal-link-1.0.1-0358d4e416ad055e1e148187987824589541f00f08ddb5855faa8ff4af619c66.a: unknown symbol `abl_link_create'
  -- toy: ^^ Could not load 'tidalzmlinkzm1zi0zi1zm0358d4e416ad055e1e148187987824589541f00f08ddb5855faa8ff4af619c66_SoundziTidalziLink_hello_closure', dependency unresolved. See top entry above.
  runStmt "hello"

This time the unmangled symbols are:

• asio::detail::keyword_tss_ptr<asio::detail::call_stack<asio::detail::thread_contextasio::detail::thread_info_base>::context>::value_
• abl_link_create
• tidal-link-1.0.1-0358d4e416ad055e1e148187987824589541f00f08ddb5855faa8ff4af619c66_Sound.Tidal.Link_hello_closure

The asio::...::value_ symbol is the same, which makes me think that this is the cause of all the other missing symbols.

[gelisam]

I see you've added a tidal-link-1.0.1 exporting an extra hello :: IO () action. Thanks, this helps to eliminate a whole bunch of code

My mistake, now that I am taking a look at the source for hello, I see that it's actually exercising the library's other functions, it's not just hello = putStrLn "hello" as I expected from the name.

[gelisam]

If I change the implementation to hello = putStrLn "hello", I still get the error, so we're back to "Thanks, this helps to eliminate a whole bunch of code" :)

[gelisam]

Deleting the rest of the hsc file makes the error disappear, so it's definitely the lazy-loading of this module which is triggering the errors.

[gelisam]

Here's a minimal hsc module which still reproduces the problem:

{-# LANGUAGE ForeignFunctionInterface #-}

module Sound.Tidal.Link where

foreign import ccall "abl_link.h abl_link_create"
  create :: IO ()

hello :: IO ()
hello = print "hello"

I'm lying about the type of abl_link_create in order to minimize the imports, so create would fail at runtime when the IO action is called, but here we're not even calling it, we're calling hello. It doesn't matter whether I keep abl_link_create or any of the other foreign imports of this file. I suspect the problem is in the linked abl_link.h file.

[gelisam]

I've narrowed it down to this line:

https://github.com/tidalcycles/Tidal/blob/a9208569cbd86ee0c0c39052f354ee4984d701d9/tidal-link/link/extensions/abl_link/src/abl_link.cpp#L21

I have simplified the .h, .cpp, and .hsc files to their bare minimum:

-- tidal-link/src/hs/Sound/Tidal/Link.hsc
{-# LANGUAGE ForeignFunctionInterface #-}

module Sound.Tidal.Link where

foreign import ccall "abl_link.h abl_dummy"
  dummy :: IO ()

hello :: IO ()
hello = print "hello"

// tidal-link/link/extensions/abl_link/include/abl_link.h
#ifdef __cplusplus
extern "C"
{
#endif // __cplusplus

  void abl_dummy();

#ifdef __cplusplus
} // extern "C"
#endif // __cplusplus

// tidal-link/link/extensions/abl_link/src/abl_link.cpp
#include <abl_link.h>
#include <ableton/Link.hpp>

extern "C"
{
  void abl_dummy()
  {
    return;
  }
}

The error disappears if I remove the #include <ableton/Link.hpp> line.

Note that cabal doesn't automatically recompile when I change those C++ files, so I had to delete the dist-newstyle folder after each change.

[gelisam]

~Narrowed it down further to~

~https://github.com/tidalcycles/Tidal/blob/a9208569cbd86ee0c0c39052f354ee4984d701d9/tidal-link/link/include/ableton/Link.hpp#L25~

~That is, I'm now hopping from ableton/Link.hpp to ableton/platforms/Config.hpp...~

[gelisam]

~...to tidal-link/link/include/ableton/platforms/asio/Context.hpp...~

[gelisam]

~...to tidal-link/link/include/ableton/platforms/asio/AsioWrapper.hpp...~

[gelisam]

~...to tidal-link/link/modules/asio-standalone/asio/include/asio.hpp...~

[gelisam]

~...to tidal-link/link/modules/asio-standalone/asio/include/asio/compose.hpp...~

[gelisam]

~...to tidal-link/link/modules/asio-standalone/asio/include/asio/impl/compose.hpp...~

[gelisam]

~...to tidal-link/link/modules/asio-standalone/asio/include/asio/associated_executor.hpp...~

[gelisam]

~...to tidal-link/link/modules/asio-standalone/asio/include/asio/system_executor.hpp...~

[gelisam]

~...to tidal-link/link/modules/asio-standalone/asio/include/asio/execution.hpp...~

[gelisam]

~This is getting pretty ridiculous. As I dove down, the overall shape of the error message has stayed the same, but the specific symbols I am missing have changed. they are now:~

~1. asio::system_category()::instance 2. abl_dummy 3. tidal-link-1.0.1-inplace_Sound.Tidal.Link_hello_closure~

~This first symbol is defined here:~

~https://github.com/tidalcycles/Tidal/blob/a9208569cbd86ee0c0c39052f354ee4984d701d9/tidal-link/link/modules/asio-standalone/asio/include/asio/impl/error_code.ipp#L196-L200~

~Rather than continuing to navigate the ridiculous indirection levels of the asio library until I reach that file, perhaps I can try to create a minimal example using just that definition?~

[gelisam]

Wait, there is another big difference in the error message: the first error message now says unknown symbol blah instead of ELF TLSGD relocation for symbol blah not supported on the target platform. I thus suspect that I made the following mistake somewhere along that long dive through the asio source code:

1. in order to minimize a given source file, I deleted some code and checked whether the hint code succeeded or failed with a linking error.
2. one of those deletions deleted the implementation of asio::system_category()::instance but kept its declaration.
3. the overall shape of the error message still looked the same, I didn't spot that the error message now said unknown symbol blah, so I continued diving down when I shouldn't have.

Thus, I now need to backpedal until the error message becomes not supported on the target platform again.

[gelisam]

Backpedaling aaaall the way back to the very first comment, I am sorry to say that I cannot reproduce the error as written: on the given code, I get unknown symbol _ZN4asio6detail15keyword_tss_ptrINS0_10call_stackINS0_14thread_contextENS0_16thread_info_baseEE7contextEE6value_E, not ELF TLSGD relocation for symbol _ZN4asio6detail15keyword_tss_ptrINS0_10call_stackINS0_14thread_contextENS0_16thread_info_baseEE7contextEE6value_E not supported on the target platform!

Perhaps we are simply using different compilers or different operating systems, and that translates to slightly different error messages for the same underlying issue? Could you check if you still get not supported on the target platform if you use the minimal Link.hsc, abl_link.h and abl_link.cpp from my comment above?

$ uname -a
Linux monolith 5.18.13-200.fc36.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Jul 22 14:03:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

$ g++ --version
g++ (GCC) 12.1.1 20220507 (Red Hat 12.1.1-1)
Copyright (C) 2022 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

$ ghc --version
The Glorious Glasgow Haskell Compilation System, version 8.10.7

$ cabal --version
cabal-install version 3.6.0.0
compiled using version 3.6.1.0 of the Cabal library 

[Zalastax]

Herculean effort @gelisam! Big appreciation from me!

Even though the issue can't be reproduced exactly, would it still make sense to fix the new error that you get? It's still valid code that we'd prefer hint to handle, right? And with a bit of luck, it would solve the original error as well?

C++ from Haskell is quite niche so I also understand if it's too much of an effort to support.

[polymorphicengine]

Could you check if you still get not supported on the target platform if you use the minimal Link.hsc, abl_link.h and abl_link.cpp from my comment above?

i also get the unkown symbol error after making those changes:

unknown symbol '_ZGVZN4asio15system_categoryEvE8instance' and unknown symbol 'abl_dummy'

[polymorphicengine]

I just tried to run the minimal program that produces the error inside of cabal repl and this works fine while cabal run still fails, don't really know what to make of it though but it seems strange

[gelisam]

Aha! You have solved the case :)

Previously, there mystery was: "why does import Sound.Tidal.Link; createSessionState work in ghci but runInterpreter $ do { setImports "Sound.Tidal.Link"; runStmt "createSessionState" } fail". That framing of the problem made it look like ghci somehow loaded libraries differently than hint, which seems weird because neither has custom library-loading code, they both delegate to the ghc API for that.

Now, the mystery is: "why does runInterpreter $ do { setImports "Sound.Tidal.Link"; runStmt "createSessionState" } work in cabal repl but fail in cabal run". Easy: because cabal is calling ghc -i with a ton of arguments specifying where to find the dependencies, whereas cabal run calls your executable, which then calls unsafeRunInterpreterWithArgs with zero arguments.

Thus, the bug is not in hint's code, but in hint's documentation. The next steps are:

1. to run cabal repl -v to see which arguments cabal is passing to ghc -i
2. to pass them all to unsafeRunInterpreterWithArgs to confirm that this works
3. to find the subset which is required to make the code succeed
4. to understand why
5. to add a section in hint's documentation explaining which extra arguments need to be passed when using a dependency which uses the ffi

[polymorphicengine]

i get the following options shown by cabal repl -v

-fbuilding-cabal-package 
-O0 
-outputdir /home/martin/Schreibtisch/hint-try/dist-newstyle/build/x86_64-linux/ghc-9.2.3/hint-try-0.1.0.0/x/hint-try/build/hint-try/hint-try-tmp 
-odir /home/martin/Schreibtisch/hint-try/dist-newstyle/build/x86_64-linux/ghc-9.2.3/hint-try-0.1.0.0/x/hint-try/build/hint-try/hint-try-tmp 
-hidir /home/martin/Schreibtisch/hint-try/dist-newstyle/build/x86_64-linux/ghc-9.2.3/hint-try-0.1.0.0/x/hint-try/build/hint-try/hint-try-tmp 
-stubdir /home/martin/Schreibtisch/hint-try/dist-newstyle/build/x86_64-linux/ghc-9.2.3/hint-try-0.1.0.0/x/hint-try/build/hint-try/hint-try-tmp 
-i 
-i/home/martin/Schreibtisch/hint-try/dist-newstyle/build/x86_64-linux/ghc-9.2.3/hint-try-0.1.0.0/x/hint-try/build/hint-try/hint-try-tmp 
-iapp 
-i/home/martin/Schreibtisch/hint-try/dist-newstyle/build/x86_64-linux/ghc-9.2.3/hint-try-0.1.0.0/x/hint-try/build/hint-try/autogen 
-i/home/martin/Schreibtisch/hint-try/dist-newstyle/build/x86_64-linux/ghc-9.2.3/hint-try-0.1.0.0/x/hint-try/build/global-autogen 
-I/home/martin/Schreibtisch/hint-try/dist-newstyle/build/x86_64-linux/ghc-9.2.3/hint-try-0.1.0.0/x/hint-try/build/hint-try/autogen 
-I/home/martin/Schreibtisch/hint-try/dist-newstyle/build/x86_64-linux/ghc-9.2.3/hint-try-0.1.0.0/x/hint-try/build/global-autogen 
-I/home/martin/Schreibtisch/hint-try/dist-newstyle/build/x86_64-linux/ghc-9.2.3/hint-try-0.1.0.0/x/hint-try/build/hint-try/hint-try-tmp 
-optP-include 
-optP/home/martin/Schreibtisch/hint-try/dist-newstyle/build/x86_64-linux/ghc-9.2.3/hint-try-0.1.0.0/x/hint-try/build/hint-try/autogen/cabal_macros.h 
-hide-all-packages 
-Wmissing-home-modules 
-no-user-package-db 
-package-db /home/martin/.cabal/store/ghc-9.2.3/package.db 
-package-db /home/martin/Schreibtisch/hint-try/dist-newstyle/packagedb/ghc-9.2.3 
-package-db /home/martin/Schreibtisch/hint-try/dist-newstyle/build/x86_64-linux/ghc-9.2.3/hint-try-0.1.0.0/x/hint-try/package.conf.inplace 
-package-id base-4.16.2.0 
-package-id hint-0.9.0.6-92428822c9fdeb18d85ea152a7ba652b507040565af42e0d670fc6384af1728a 
-XHaskell2010

i tried including all of them as arguments in unsafeRunInterpreterWithArgs (except the ones about the package-db), but get the same error.

what kindof jumps into my eye is

-optP-include 
-optP/home/martin/Schreibtisch/hint-try/dist-newstyle/build/x86_64-linux/ghc-9.2.3/hint-try-0.1.0.0/x/hint-try/build/hint-try/autogen/cabal_macros.h 

in the ghc-docs for optP it says makes sense only if -cpp is also on, but also no luck with that...