Implementing support for `go-embed-python`

[mnaser]

trafficstars

Do you think there's an easy way to be able to use this project in combination with https://github.com/kluctl/go-embed-python which would help us make a fully independent golang binary? :)

[codablock]

fyi, I assume go-ansible might also be interested in https://github.com/kluctl/go-jinja2, which uses the mentioned library to embed Jinja2 as well. The API is far from perfect, but I'm happy to accepts PRs in that regard.

[apenella]

Hi, @mnaser! Thank you very much for opening that issue. I hadn't come across go-embed-python before, and it looks like it could open a multitude of possibilities. Big kudos to @codablock for the excellent work!

It seems your idea is to develop an application that embeds both Python and the Ansible playbook, inventory file, etc., needed for execution in a single binary, isn't it?

As I delved into understanding go-embed-python, I also explored go-jinja2. One possible approach for your application could be utilizing the pip helpers provided by go-embed-python to install Ansible on the embedded Python. Subsequently, using embed_utils, you could incorporate your Ansible code. When configuring the Ansible playbook Cmd, consider using the files loaded within the embedded files. @codablock Do you think it is the correct approach?

While I don't see the need to include this as a feature in the go-ansible library, I aim to keep the library focused on executing Ansible commands. If feasible, I'd be keen to develop an example showcasing both libraries working together. It could be interesting for many and open up numerous possibilities.

Thank you both!

[codablock]

@apenella That sounds like something that should work. I also just realised that I misunderstood this project...my understanding was that it tries to re-implement Ansible in Go and thus needs Jinja2 support. Now I understand that go-ansible is a wrapper library around the original Ansible, in which case what you describe is the way to go.

[apenella]

@codablock, thank you very much for your confirmation!

@mnaser Knowing that, do you feel your doubt has been resolved? As mentioned earlier, I plan to create an example showing the integration of both libraries. However, this task is not on my immediate agenda.

[ditwrd]

Hello there, this issue prompted me to create a CLI side project that leverages the "embed ansible" idea.

TL;DR: Tried copying what go-jinja2 do, it work with ansible, but issue in binary size due to the sheer amount of collections, can be mitigated with installing ansible-core instead (collections installed separately)

I attempted to replicate the folder structure exactly as in https://github.com/kluctl/go-jinja2, and it worked. However, I encountered a problem with the binary size. For the ansible package, the generated Python modules resulted in a substantial 1.7GB of files (all platform) and yield a 351MB binary file (seems that it only embed modules for the current build platform).

To address this size issue, I opted for ansible-core (93MB py modules, yield 77MB binary) and installed necessary collections through ansible-galaxy. Perhaps a generated requirements.yml could assist in managing these collections. Due to this limitation, for anyone wanting to "embed ansible" to their app, I suggest employing a strategy similar to how games are delivered nowadays—initial "download" upon installation. This way, we can create a binary as small as possible to facilitate distribution and allow end users to "download" related dependencies. Maybe there are some other way to make the binary smaller, but 77MB for a standalone embed ansible binary, quite an improvement from "install it using pip" approach, might be useful in some cases.

I haven't delved too deeply into this yet, as it's my first Golang project, and I only started it an hour ago. Please note that my testing has been limited to ansible version and ansible galaxy collection install. I haven't tested with a literal ansible playbook. However, given that the approach of extracting things to /tmp, I believe it should work (at least, I hope so).

I hope this clarifies the "what if" scenario for other people seeing this issue. I'll provide more information if any issues arise. While I'm unsure if this falls within the scope of this golang library and issue, perhaps the information above can be useful.

[apenella]

Hi @ditwrd! Thank you very much for sharing your insights on this. I am considering writing an example related to that because I felt appealed by this idea. I believe your experience will be very helpful for anyone. I will reach out to you if I need some assistance on this.

Have you published that project publicly?

[ditwrd]

Yes @apenella, it's public and you can check it here

The project is practically still a barebone (weekend only fun little project you might say), I just merge my PR for the embed Ansible POC (kudos @codablock for the go-jinja2 project), just run the main.go file and it will:

• Check the embedded ansible version
• List all of the collections (this should be empty)
• Install community.docker to the collection
• List the collection again (community.docker and its dependencies should be there)

I guess the next thing I should do is to see if it is possible to trigger the embed ansible via go-ansible, I've never used go-ansible before, maybe I'll learn something along the way

[apenella]

@mnaser @ditwrd @codablock Here you can see an example using the go-ansible library along with go-embed-python. I hope it could inspire you and someone else to create new solutions!

[apenella]

An example is available in https://github.com/apenella/go-ansible/releases/tag/v2.0.0