idea: Go bindings **Native** support without CGO

[yuchanns]

Hello community!

I'd like to introduce you to a native method for supporting Go bindings without CGO.

This capability primarily relies on two libraries: purego and libffi.

• purego: A library for calling C functions from Go without Cgo, but it lacks support for passing and returning structure values.
• libffi: A portable foreign-function interface library. It is also a dependency of Python and Ruby for their FFI functionality.

To support the passing and returning of structure values, we need to create lightweight Go libffi bindings by wrapping purego, which is already provided by ffi. We can either use it or maintain it ourselves directly.

Then we can call arbitrary C functions using libffi from Go without enabling CGO, which is why I refer to it as Native support!

Here's a POC that demonstrates its functionality with backend memory and aliyun_drive: gopendal.

Pros:

• No need to enable CGO. - It is beneficial to promote opendal to various well-known pure Go basic libraries.
• Dynamic injection of libopendal_c.so - Users can inject custom featured libs under their control.

Cons:

• Still requires libopendal_c.so. - We can embed it in go files for each platform_suffix file. eg: *_linux.go, *_windows.go,
• Depends on preinstalled libffi. - libffi is widely used in many languages and commonly preinstalled on many platforms. You can easily install it with the package manager if not available.
• Currently only supports Linux and BSD. - This limitation arises from the go wrapper ffi. Ideally, it should be able to support platforms where libffi and purego intersect, meaning Windows, Linux, and MacOS should all be supported.
• Performance has not been tested yet. But I don't think this will be a concern as CGO is the major problem.

What do you think? I look forward to your comments!

[yuchanns]

Source: https://github.com/ebitengine/purego/issues/236

[Xuanwo]

Adding some context:

• https://github.com/apache/opendal/issues/1472
• https://github.com/apache/opendal/issues/1366

We have discussed similiar ideas in the past.

---

• Still requires libopendal_c.so. - We can embed it in go files for each platform_suffix file. eg: *_linux.go, *_windows.go,

From the ASF perspective, distributing binaries in source form is not permitted. Therefore, we might not include in this repo directly. Perhaps we could set up an additional repository for releases.

[yuchanns]

From the ASF perspective, distributing binaries in source form is not permitted.

I add a pros: Dynamic Inject distributing binaries. It can be an advantage as current features of bindings can not be controlled by users.

[Xuanwo]

• Depends on preinstalled libffi. - libffi is widely used in many languages and commonly preinstalled on many platforms. You can easily install it with the package manager if not available.

Confirmed. libffi is widely pre-installed and depended:

https://archlinux.org/packages/core/x86_64/libffi/

[zjregee]

Looking forward to this!

[yuchanns]

Adding some context:

Ok, I've checked the context, and identified some major blockers:

• Distributing binaries in the source is not permitted.
• Binary size All-in-one.
• go-import doesn't support subdirectory.

Well, these are truly obstacles. I've no idea how to resolve them.

TBH, I do think it is impossible to support subdirectories in the foreseeable future as the Go team prioritizes things based on their needs. Even if the problem has been resolved, we still have to challenge the binary distribution as Go is not designed to be extended with other languages.

IMO, it is inevitable to have multiple root repos for go bindings of various features. As long as these sources are distributed under opendal.apache.org/go, users can trust them, and no need to care where are they placed.

As for the *.so binaries, they can be loaded separately just like various dialects in gorm.io/driver/{sqlite,mysql,postgres}.

That is to say:

package main

import (
	opendal "opendal.apache.org/go"
	_ "opendal.some.others/services/s3" // this dynamic inject *.so
)

I regret that these guideline-level issues have prevented the implementation of go-binding.

[Xuanwo]

package main

import (
	opendal "opendal.apache.org/go"
	_ "opendal.some.others/services/s3" // this dynamic inject *.so
)

Wow, I didn't know that was possible! I think we can start with github.com/apache/opendal/bindings/go and your own "*.so" files to see how it functions. If it works well, I'm willing to help manage the artifacts by setting up a new repository called github.com/apache/opendal-go-artifacts (just for example).

[yuchanns]

I think we can start with github.com/apache/opendal/bindings/go and your own "*.so" files to see how it functions.

Cool. I will take a try next week.

[Xuanwo]

Thank @yuchanns for make this idea heppen! Would you like to create a new tracking issues for our next plan?

[yuchanns]

Added https://github.com/apache/opendal/issues/4892

[Xuanwo]

This idea has been advanced at https://github.com/apache/opendal/issues/4892