Module system suggestions

[sbarisic]

Besides some way to load other modules written in gravity, i would suggest implementing the functionality to load native libraries as modules too.

For example module "test" would load test.gravity but in case that does not exist it would try test.dll or test.so and load a global gravity_load function where you can register functions/classes using the native API.

[bates64]

import "test" # module named test. See issue #37
import "test.gravity"
import "test.so"
import "test.dll"
import "./test" # looks for test.gravity, then test.dll, then test.so?

[sbarisic]

Yes, except i would make the file extensions optional and only resolve native libraries after nothing written in gravity has been found. That way the language can be extended with system dependent functionality on different systems and the common gravity codebase wouldn't have to change.

[bates64]

@cartman300 yep, that's what I was getting at with import "./test".

[RitamDey]

Will exposing native C API be really good?? Should Gravity allow external code to randomly poke at a running interpreter??? Nobody developing a new modern language seems to be doing it!!!! Swify doesn't does it nor does Go neither does PyPy!!! It will basically make breaking changes in the interpreter very hard!!! Exposing C API has been one of major causes why Tracing GC is not implemented in CPython!! Also exposing GC is probably not good as the native extension writer may fail to comply with GC standards!! Please don't expose native API

[Mijyuoon]

@GreenJoey So you're basically saying that native modules shouldn't be supported because some developers are retards or what? Also neither Swift nor Go are interpreted languages so that part of your comment is meaningless.

[sbarisic]

Literally every language you posted as example supports native code interop in one way or another.

[RitamDey]

Go doesn't let you call C directly but via a built-in C FFI. PyPy also does allow C code but via C FFI. And you can't assume that every programmer is perfectly normal. I am suggesting developers not to support native modules but support calling C functions via FFI.

[marcobambini]

I like the idea to add support for FFI (in addition to the existing C API). In this way users will be able to call c code specifying its prototype from within Gravity.

[parro-it]

import "test" # module named test. See issue #37 import "test.gravity" import "test.so" import "test.dll" import "./test" # looks for test.gravity, then test.dll, then test.so?

@nanalan we should also need a file extension for json bytecode, I think gravity should load also pre-compilend, non-native modules.

[bates64]

we should also need a file extension for json bytecode

Neko's bytecode is .n - we could use .g for Gravity bytecode?

[marcobambini]

Yes we really need a file extension for json bytecode... I was thinking about .gcode and some user suggested .gr

[sbarisic]

@marcobambini Please don't use .gcode, this may be confused with G code for CNC machines.

@GreenJoey An interpreted programming language can not have a foreign function interface (at least not without doing magic), marshalling data structures from Gravity side to native side and back isn't child's play, especially if you try implementing it in a language like C.

This isn't even that simple in a programming language which has runtime reflection like C#, see https://github.com/cartman300/Lua.NET/blob/master/Lua.NET/Advanced.cs#L207-L236

There's also a reason there's FFI for LuaJIT, not the interpreted one. There are binary modules for the original. http://lua-users.org/wiki/LuaBinaryModules

[Mijyuoon]

@GreenJoey Basically the only difference between FFI and binary modules in this case is where the actual API is defined.

In case of FFI you have to make wrappers around native functions in your target language so it all doesn't feel clunky to use.

In case of native modules it's basically same stuff except that you define APIs in native code. So unless the way of doing it which the target language provides is total garbage there's nothing much that you can do wrong. Because the majority of screw ups will be in code that actually does what you want to do, not in binding/interfacing/glue/whatever code.

[marcobambini]

This interesting thread contains different proposals/discussions, I wonder if there is a way to split discussions and merge other similar threads:

• Module subsystem implementation (to be merged with https://github.com/marcobambini/gravity/issues/37)
• FFI/native code support
• Official extensions (.gravity for source code and something else for compiled code)

[kazzkiq]

I like the .g extension. Its simple, resembles C file extensions, and apparently have no relevant programming langs/apps using it as of today. Also not picked up by anyone on Github language files. Seems like a good choice.

[fnky]

I thought about this and propose some exporting functionality (similar to ECMAScript) within Gravity code:

// a.gravity
export default class Vector { /* ... */ }
export class Point { /* ... */ }
export func dot (v1, v2) { /* ... */ }

// b.gravity
import Vec, { Point, dot } from './a'  // Import a.gravity relative to this file (b.gravity)

func main () {
  var v1 = Vector(1, 1, 1)
  var v2 = Vector(0.5, 0.5, 0.5)
  var v3 = dot(v1, v2)
}

Where Vector is imported as Vec and other export names are the same.

and use import for Gravity files, and module for native modules and/or libraries.

module "fs" // bytecode or external library?
module "./math" use { random } // relative bytecode/library?
import { filter, map } from "gravity-func" // packages/gravity-func/gravity-func.gravity
import Vector from "./vector" // import relative gravity file

func main () {
  var fHandle = readdir("./tmp") // readdir() exposed from fs module
  var rand = random.nextInt()
  var evens = filter(func (x) { return (x % 2) === 0 }, [1, 2, 3, 4])
  var v = Vector()
}

In cases where main function is defined in an imported module, either expose it as a function or explicitly raise an error to tell the user that main must only be defined in top-level file (similar to Swift).

[parro-it]

@marcobambini a wiki page could be useful. Or else, a Pr that start to fill the "internals" doc section files?

[parro-it]

@fnky I link the syntax, is very similar to es6...

import { filter, map } from "gravity-func" // packages/gravity-func/gravity-func.gravity

I think we need a way to disambiguate the package name. If the package manager for gravity will use (as all seems prefer) a multi-service register, the package should probably imported as something like

import { filter, map } from "github/parro-it/gravity-func"

and use import for Gravity files, and module for native modules.

What is the advantage of using two different syntaxes?

[fnky]

1. For multi service it could be something like

import { filter, map } from "github:parro-it/gravity-func"

for known services like github, bitbucket etc. and

import { filter, map } from "[email protected]:parro-it/gravity-func"

for private services, which aren't known to Gravity.

2. The advantage is to differentiate native modules from source files. Native modules would/could have their functions, variables, classes, etc. exposed without explicitly importing them like import, whereas imports would require functions and variables to be exported from that file/package.

[marcobambini]

In the case of functionality of loading native libraries as modules how you would declare symbols to load?

[fnky]

I'm not quite sure what you mean specifically, but let me try to explain my idea:

In case of loading a native library, upon importing it:

module "audio"

func main () {
  var h = Audio("./foo.wav") // class exposed from `audio` library.
  h.play()
}

or specifying what to use from the module

// same as module "audio" use { AudioDecoder }
module "libaudio" use { AudioDecoder }

func main () {
  var h = Audio() // this should fail, since we don't expose it because of `use`.
}

If the defined module to load isn't relative (starting with ./) it will:

1. Search in paths, defined in PATH and GRAVITY_MODULE_PATH for example, where GRAVITY_MODULE_PATH will have higher precedence than PATH, if defined.
2. Search for symbols/libraries with the given name—in this case audio—as well as files prefixed with lib* (e.g. libaudio).

If the defined path for the module is a relative path `./lib/audio' it should do step 2. but in the given directory of the module.

You could also allow users to specify a search path to the command-line, which will have higher precedence than PATH and GRAVITY_MODULE_PATH

# setting gravity module search path to search in ./lib and ./thirdparty
GRAVITY_MODULE_PATH=./lib:./thirdparty gravity # ...

# same as above, but through the command-line
# will have higher precedence over GRAVITY_MODULE_PATH and PATH
gravity --module-path ./lib --module-path ./thirdparty

My current terminology is module for native libraries and satellites, packages and file for Gravity source packages and source files.

[marcobambini]

So .g for Gravity bytecode?

[YurySolovyov]

I think module system should also provide some isolation mechanism, because with current #include, everything gets pulled into a global namespace -> name collisions, namespace pollution.

[marcobambini]

@YurySolovyov yes I agree and I currently working on a new module subsystem for Gravity.