feat: support more file protocols support in `include`

[Zheaoli]

Description

---

Message from the maintainers:

Myself, I meet some circumstances that are not supported by the end. Here's the detail

the envd allows people to use include function to include the existing library to reuse the code. In my circumstance, I don't want to make my code public so that the code repo would be private. But for now, the envd include mechanism supports pure HTTP and HTTP git only.

So maybe we can support more file protocols in include. I think we can use the URL scheme to identify different file protocols. For example:

1. http://
2. git+http://
3. gti+ssh://
4. s3://
5. ....

[kemingy]

SGTM. We should support more protocols for the include function.

By the way, we may need an abstract file layer for the code and data. Things include:

• authentication
• storage
• sync

[Zheaoli]

maybe I can make a proposal with design detail

[gaocegege]

@Zheaoli Thanks! Looking forward to it.

[Xuanwo]

Hi, I'm the maintainer of OpenDAL: the Open Data Access Layer to access data freely, painlessly, and efficiently. I am interested in exploring ways to combine envd with opendal to make MLOps with various storage backends more easily.

I have some quick notes about this issue:

• include is required to be implemented in golang?
  • Maybe we can integration will rclone directly.
• Does envd allow implementing plugins in other languages like rust (the one used by opendal)?

[gaocegege]

/cc @kemingy

include is required to be implemented in golang?

For now, yes.

Does envd allow implementing plugins in other languages like rust (the one used by opendal)?

There is no such feature now.

[Xuanwo]

I am researching on https://github.com/datafuselabs/opendal/issues/1366. Maybe we can build an opendal Go binding with purego?

If this is acceptable, I plan to start supporting s3 first, without touching (or breaking) the existing features we have supported so far.