Add Scala Native Examples

[c0d33ngr]

This is a draft to close issue #3647

[c0d33ngr]

Hi @lihaoyi something like this or I have to use scalatags and mainargs?

[lihaoyi]

Let's use the same scalatags/mainargs example as we did in scalalib/basic/1-simple/.

You can get rid of most of the docs that are already part of scalalib/basic/1-simple/. The point of this page is to show what's unique about scala native, so the docs should highlight the differences and unique features of it: e.g. the compile to binary, the different optimization levels, etc.

[c0d33ngr]

Alright, it's noted

[c0d33ngr]

//build.mill
package build
import mill._, scalalib._, scalanativelib._

object `package` extends RootModule with ScalaNativeModule {
  def scalaVersion = "2.13.11"
  def scalaNativeVersion = "0.5.5"
  def nativeLinkingOptions = pathToWhereTheTargetWasSaved

  object test extends ScalaNativeTests {
    def ivyDeps = Agg(ivy"com.lihaoyi::utest:0.8.4")
    def testFramework = "utest.runner.Framework"
  }
}

#makefile
all : 
	gcc -m64 -shared -c native-src/HelloWorld.c -o target/libstub.so

@lihaoyi good day, I've some few questions to ask

Is it possible for me to run the commands in makefile from build.mill config? Also how get the full file path of where am currently working in?

When I replace the def nativeLinkingOptions with the full path to the choice location to store the .so file, it worked. But before that I had to run make from terminal to generate the .so file in the location

That's why I asked whether it's possible to run the comments in makefile from build.mill

Lastly, is this what you want for the 2-interop code example?

[lihaoyi]

Is it possible for me to run the commands in makefile from build.mill config?

Sure, os.call(("make, "blah")). Though I suggest if you know what you want to call, you can os.call gcc directly rather than going through make

Also how get the full file path of where am currently working in?

T.dest or millSourcePath depending on what where am currently working in means

When I replace the def nativeLinkingOptions with the full path to the choice location to store the .so file, it worked. But before that I had to run make from terminal to generate the .so file in the location

Make an upstream task generate the SO file and return a PathRef for the downstream task to consume

Lastly, is this what you want for the 2-interop code example?

Yes the 2-interop example looks great. Perhaps let's make it return a value as well, so rather than printString we make it reverseString or something, just so people can see how that works? Other than that it looks great: a tiny C program, a tiny Scala facade, and a tiny Scala program all interacting. You just need to wire up gcc to compile the thing as part of Mill so we can avoid having to run make manually

[c0d33ngr]

Good day @lihaoyi please you mind looking through 1-simple and 2-interop examples if it's in line with what you want cause they ran successfully with mill run? The issue i have with them is that my tests for both of them seem to have problem with linking. The error output for 1-dimple below

When i had linking error for the mill run hello for 1-simple example, it was becsuse i didn't include a nstiveLinkingOptions in build.millbut for mill test I'm a little bit confused there

#1 [136/141] test.compile 
#1 [info] compiling 1 Scala source to /workspaces/milling-two/sunday-work/mill/example/scalalib/native/1-simple/out/test/compile.dest/classes ...
#1 [info] done compiling
#2 [141/141] test.nativeLink 
#2 [info] Linking (multithreadingEnabled=true, disable if not used) (922 ms)
#2 [info] Discovered 858 classes and 5426 methods after classloading
#2 [error] Found 2 unreachable symbols!
#2 [error] Unknown static method scala.scalanative.testinterface.TestMain.main(java.lang.String[]): scala.runtime.BoxedUnit, referenced from:
#2 
#2 
#2 [error] Unknown type scala.scalanative.testinterface.TestMain, referenced from:
#2 
#2 
#2 [info] Total (987 ms)
1 targets failed
test.nativeLink scala.scalanative.linker.LinkingException: Unreachable symbols found after classloading run. It can happen when using dependencies not cross-compiled for Scala Native or not yet ported JDK definitions.

Also i would love to know ehether mainargs works with Scala Native as I don't want to use fansi as a third-party dependency cause i used it in 1-simple example. i got this error below.

Linking Exception
#2 [138/138] foo.nativeLink
#2 [info] Linking (multithreadingEnabled=true, disable if not used) (2814 ms)
#2 [info] Discovered 873 classes and 5571 methods after classloading
#2 [error] Found 19 unreachable symbols!
#2 [error] Unknown type mainargs.MainData, referenced from:
#2 method foo.Foo$.main(java.lang.String[]): scala.runtime.BoxedUnit at Foo.scala:21
#2 static method foo.Foo.main(java.lang.String[]): scala.runtime.BoxedUnit at Foo.scala:21
#2
#2
#2 [error] Unknown type mainargs.ArgReader$Simple, referenced from:
#2 method foo.Foo$.main(java.lang.String[]): scala.runtime.BoxedUnit at Foo.scala:21
#2 static method foo.Foo.main(java.lang.String[]): scala.runtime.BoxedUnit at Foo.scala:21
#2
#2
#2 [error] Unknown type mainargs.ArgReader, referenced from:
#2 method foo.Foo$.main(java.lang.String[]): scala.runtime.BoxedUnit at Foo.scala:21
#2 static method foo.Foo.main(java.lang.String[]): scala.runtime.BoxedUnit at Foo.scala:21
#2
#2
#2 [error] Unknown type mainargs.arg, referenced from:
#2 method foo.Foo$.main(java.lang.String[]): scala.runtime.BoxedUnit at Foo.scala:21
#2 static method foo.Foo.main(java.lang.String[]): scala.runtime.BoxedUnit at Foo.scala:21
#2
#2
#2 [error] Unknown type mainargs.MethodMains, referenced from:
#2 method foo.Foo$.main(java.lang.String[]): scala.runtime.BoxedUnit at Foo.scala:21
#2 static method foo.Foo.main(java.lang.String[]): scala.runtime.BoxedUnit at Foo.scala:21
#2
#2
#2 [error] Unknown type mainargs.ArgSig, referenced from:
#2 method foo.Foo$.main(java.lang.String[]): scala.runtime.BoxedUnit at Foo.scala:21
#2 static method foo.Foo.main(java.lang.String[]): scala.runtime.BoxedUnit at Foo.scala:21
#2
#2
#2 [error] Unknown constructor mainargs.arg(java.lang.String, char, java.lang.String, bool, bool), referenced from:
#2 method foo.Foo$.main(java.lang.String[]): scala.runtime.BoxedUnit at Foo.scala:21
#2 static method foo.Foo.main(java.lang.String[]): scala.runtime.BoxedUnit at Foo.scala:21
#2
#2
#2 [error] Unknown constructor mainargs.MethodMains(scala.collection.immutable.Seq, scala.Function0), referenced from:
#2 method foo.Foo$.main(java.lang.String[]): scala.runtime.BoxedUnit at Foo.scala:21
#2 static method foo.Foo.main(java.lang.String[]): scala.runtime.BoxedUnit at Foo.scala:21
#2
#2
#2 [error] Unknown type mainargs.arg$, referenced from:
#2 method foo.Foo$.main(java.lang.String[]): scala.runtime.BoxedUnit at Foo.scala:21
#2 static method foo.Foo.main(java.lang.String[]): scala.runtime.BoxedUnit at Foo.scala:21
#2
#2
#2 [error] Unknown type mainargs.main, referenced from:
#2 method foo.Foo$.main(java.lang.String[]): scala.runtime.BoxedUnit at Foo.scala:21
#2 static method foo.Foo.main(java.lang.String[]): scala.runtime.BoxedUnit at Foo.scala:21
#2
#2
#2 [error] Unknown type mainargs.TokensReader$StringRead$, referenced from:
#2 method foo.Foo$.main(java.lang.String[]): scala.runtime.BoxedUnit at Foo.scala:21
#2 static method foo.Foo.main(java.lang.String[]): scala.runtime.BoxedUnit at Foo.scala:21
#2
#2
#2 [error] Unknown constructor mainargs.ParserForMethods(mainargs.MethodMains), referenced from:
#2 method foo.Foo$.main(java.lang.String[]): scala.runtime.BoxedUnit at Foo.scala:21
#2 static method foo.Foo.main(java.lang.String[]): scala.runtime.BoxedUnit at Foo.scala:21
#2
#2
#2 [error] Unknown constructor mainargs.main(java.lang.String, java.lang.String), referenced from:
#2 method foo.Foo$.main(java.lang.String[]): scala.runtime.BoxedUnit at Foo.scala:21
#2 static method foo.Foo.main(java.lang.String[]): scala.runtime.BoxedUnit at Foo.scala:21
#2
#2
#2 [error] Unknown type mainargs.TokensReader, referenced from:
#2 method foo.Foo$.main(java.lang.String[]): scala.runtime.BoxedUnit at Foo.scala:21
#2 static method foo.Foo.main(java.lang.String[]): scala.runtime.BoxedUnit at Foo.scala:21
#2
#2
#2 [error] Unknown type mainargs.ParserForMethods, referenced from:
#2 method foo.Foo$.main(java.lang.String[]): scala.runtime.BoxedUnit at Foo.scala:21
#2 static method foo.Foo.main(java.lang.String[]): scala.runtime.BoxedUnit at Foo.scala:21
#2
#2
#2 [error] Unknown type mainargs.ArgSig$, referenced from:
#2 method foo.Foo$.main(java.lang.String[]): scala.runtime.BoxedUnit at Foo.scala:21
#2 static method foo.Foo.main(java.lang.String[]): scala.runtime.BoxedUnit at Foo.scala:21
#2
#2
#2 [error] Unknown type mainargs.ArgReader$, referenced from:
#2 method foo.Foo$.main(java.lang.String[]): scala.runtime.BoxedUnit at Foo.scala:21
#2 static method foo.Foo.main(java.lang.String[]): scala.runtime.BoxedUnit at Foo.scala:21
#2
#2
#2 [error] Unknown type mainargs.MainData$, referenced from:
#2 method foo.Foo$.main(java.lang.String[]): scala.runtime.BoxedUnit at Foo.scala:21
#2 static method foo.Foo.main(java.lang.String[]): scala.runtime.BoxedUnit at Foo.scala:21
#2
#2
#2 [error] Unknown type mainargs.main$, referenced from:
#2 method foo.Foo$.main(java.lang.String[]): scala.runtime.BoxedUnit at Foo.scala:21
#2 static method foo.Foo.main(java.lang.String[]): scala.runtime.BoxedUnit at Foo.scala:21
#2
#2
#2 [info] Total (2915 ms)
1 targets failed
foo.nativeLink scala.scalanative.linker.LinkingException: Unreachable symbols found after classloading run. It can happen when using dependencies not cross-compiled for Scala Native or not yet ported JDK definitions.

[lihaoyi]

@c0d33ngr i think the problem is that scala native dependencies require two pairs of ::s not just one pair

[c0d33ngr]

Could it be as a result of not returning PathRef in the build.mill after compiling the c code?

[c0d33ngr]

Okay... You mean for the mainargs double :: Noted

[c0d33ngr]

You're so right!! Thank you for spotting the source of the test errors.☺️ Let me see if I can wrap up everything today or latest tomorrow

On Wed, 9 Oct 2024 at 11:47, Li Haoyi @.***> wrote:

@.**** commented on this pull request.

In example/scalalib/native/3-multi-module/build.mill https://github.com/com-lihaoyi/mill/pull/3657#discussion_r1793296175:

+package build +import mill., scalalib., scalanativelib._

+trait MyModule extends ScalaNativeModule {

• def scalaVersion = "2.13.11"
• def scalaNativeVersion = "0.5.5"
• object test extends ScalaNativeTests {
• def ivyDeps = Agg(ivy"com.lihaoyi::utest:0.8.4")
• def testFramework = "utest.runner.Framework"
• } +}

+object foo extends MyModule {

• def moduleDeps = Seq(bar)
• def ivyDeps = Agg(ivy"com.lihaoyi::mainargs:0.4.0")

This should probably be ivy"com.lihaoyi::mainargs::0.4.0"

— Reply to this email directly, view it on GitHub https://github.com/com-lihaoyi/mill/pull/3657#pullrequestreview-2356772319, or unsubscribe https://github.com/notifications/unsubscribe-auth/AZKEA4TUZWGTZMCOS5B7CP3Z2UCVFAVCNFSM6AAAAABPJLOPOWVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDGNJWG43TEMZRHE . You are receiving this because you were mentioned.Message ID: @.***>

[lihaoyi]

Sounds good @c0d33ngr! Take your time and let me know when you'd like a review, I assume it'll take a few rounds to get it merged so we shouldn't be in any hurry

[c0d33ngr]

Okay. For this my draft PR, 1-simple works using both mill run and mill test thanks to the :: 2-interop works only using mill run. mill test still has the linking problem 3-multi-module both mill run and mill test has linking problem similar to 2-interop 4-common-config works well except that os.resource doesn't exist. I see other common-config examples and notice them trying to access the resources and custom-redources directories from mill config. I didn't do similar cause I don't know how to access mill resources directory path. How can someone get the path of resources of mill?

@lihaoyi what do you think?

[lihaoyi]

I did a review pass. Please make sure you read through the original ticket and earlier comments, it doesn't make sense for me to keep repeating things I've already written

[c0d33ngr]

Alright @lihaoyi I did the changes to the draft PR You can review 1-simple, 2-interop and 3-common-config exapmles

About 3-multi-module, it seem to be the most complicated one in the examples code. The independent module(bar) works well but the one which depends on bar which is foo seem to have linking problem. I have done lots of debugging but don't seem to see what I could be doing wrong. Step below

mill bar.nativeCompiled
SUCCESS OUTPUT MESSAGE

LD_LIBRARY_PATH=/../example/scalalib/native/3-multi/bar/target mill bar.run hello
SUCCESS OUTPUT MESSAGE

mill foo.nativeCompiled
SUCCESS OUTPUT MESSAGE

LD_LIBRARY_PATH=/.../mill/example/scalalib/native/3-multi/bar/target:/.../mill/example/scalalib/native/3-multi/foo/target:$LD_LIBRARY_PATH mill foo.run --bar-text hello --foo-text world


#1 [info] Optimizing (debug mode) (4693 ms)
#1 [info] Produced 3 LLVM IR files
#1 [info] Generating intermediate code (3848 ms)
#1 [info] Compiling to native code (7827 ms)
#1 [info] Linking with [pthread, dl, HelloWorldBar, HelloWorldFoo]
#1 [error] /usr/bin/ld: cannot find -lHelloWorldBar: No such file or directory
#1 [error] clang: error: linker command failed with exit code 1 (use -v to see invocation)
#1 [info] Total (19741 ms)
1 targets failed
foo.nativeLink scala.scalanative.build.BuildException: Failed to link /workspaces/milling-two/sunday-work/mill/example/scalalib/native/3-multi/out/foo/nativeWorkdir.dest/native/foo.Foo

I have a question to ask also, if I want to make use of a file or directory path of a module in another module how to go about it in mill

[lihaoyi]

You can read things from other modules' folders by declaring them as Task.Sources in the upstream module; you should never write things in other module's folders, as all writes should go to Task.dest.

For the scala-native thing, I have no idea. Try the scala-native discord channel?

[c0d33ngr]

You can read things from other modules' folders by declaring them as Task.Sources in the upstream module; you should never write things in other module's folders, as all writes should go to Task.dest.

For the scala-native thing, I have no idea. Try the scala-native discord channel?

I needed to know so I can direct the location of the header file foo depend on, no writing to it

[c0d33ngr]

I'm out of ideas on why ./mill test for 2-interop will have linking problem but ./mill run will run successfully. I will have to move on to the next 3-multi-module while I ask for help from the online communities. it similar error with 3-multi-module test too

[info] done compiling
[2666]   1 targets failed
[2666]   test.nativeLink scala.scalanative.build.BuildException: Failed to link /workspaces/mill-codespace/tuesday/mill/out/example/scalalib/native/2-interop/local/test.dest/sandbox/run-1/out/test/nativeWorkdir.dest/native/scala.scalanative.testinterface.TestMain

[c0d33ngr]

@lihaoyi what you think

[c0d33ngr]

@lihaoyi I've done the changes

[lihaoyi]

@lolgab could you take a pass at reviewing this? Since you have much more experience with scala native than I do

[lihaoyi]

Thanks @lolgab!

@c0d33ngr if we can just pass in the C sources via resources/scala-native, do we still need def nativeSources to reference them explicitly? Or would it suffice just to let scala-native pick them up from the resource path?

I think for the suggestion to add a third-party C library to use in the example, let's leave that to a follow-up bounty/PR, since it's a bit out of scope of the original ticket that was intentionally pretty minimal https://github.com/com-lihaoyi/mill/issues/3647

[c0d33ngr]

Thanks @lolgab!

@c0d33ngr if we can just pass in the C sources via resources/scala-native, do we still need def nativeSources to reference them explicitly? Or would it suffice just to let scala-native pick them up from the resource path?

I think for the suggestion to add a third-party C library to use in the example, let's leave that to a follow-up bounty/PR, since it's a bit out of scope of the original ticket that was intentionally pretty minimal #3647

@lolgab is so right!! As long as the C codes are in resources/scala-native, Scala Native will detect them so there's no need for compiling them within a task. I resolved most of the changes he suggested

[c0d33ngr]

Any more changes needed @lihaoyi ?

[c0d33ngr]

One nit and then I think we're good, @c0d33ngr once you fix the merge conflicts and we get a green CI run I'll merge this and close out the bounty

Alright.. It's all greeny

At first, it wasn't but all of a sudden it restarted and ran successfully. Thought, the errors doesn't seem to be about scala-native examples

[lihaoyi]

@c0d33ngr just a flaky test, not related to your PR. Thanks again for your efforts!

Email me your international bank transfer details and I will close out the bounty