[RFC] Local tools in lxbe

[PiotrZierhoffer]

@mithro The task is to enable local tools to be used by LXBE.

We'd want to allow both locally installed binaries and Docker images to be used as dependencies.

As BuildEnv grows in capabilities, I think it would make the most sense to have a configuration file for local tools. The way I see it:

local-tools.cfg:

nextpnr-ecp5 #available in $PATH
nextpnr-ice40=/home/lxbe-user/ice40tools/bin  #add this path to $PATH
ninja=/home/lxbe-user/ninja/setup-paths.sh # source this file so that $PATH is set up correctly
yosys=/home/lxbe-user/yosys/setup-docker.sh #source this file that will prepare the docker image

An alternative notation, more verbose:

local-tools.ini:

[nextpnr-ecp5] #available in $PATH

[nextpnr-ice40]
path=/home/lxbe-user/ice40tools/bin  #add this path to $PATH

[ninja]
script=/home/lxbe-user/ninja/setup-paths.sh # source this file so that $PATH is set up correctly

[yosys]
script=/home/lxbe-user/yosys/setup-docker.sh #source this file that will prepare the docker image

This notation would allow to specify version overrides or conda channel in future, if required.

Of course running/sourcing files will be done automatically by BuildEnv. Tools not available in the config will be fetched from Conda.

The local installation scenario is fairly trivial. The problem is with Docker, as there is no straightforward way to forward a process call to docker. One possible approach would be to provide a docker image along with a configuration/bootstrap script.

The script would create, for each binary exposed by the image, a wrapper script, that would simply call docker run with appropriate parameters. The scripts would be placed in build/.

For this to work we'd need to recreate a directory structure of the current lxbe instance and mount it in the docker image, along with /tmp. This would give us a sensible granularity and ease of use with different tools.

It would, of course, affect the performance, as each and every call to an exposed binary would translate to docker run. Having it more tailored to specific scenarios, e.g. building zephyr, would make the communication a bit easier, but would require support directly in lxbe, and that would not scale well (and will be difficult to maintain).

[mithro]

There are a couple of things here.

(a) We want to work out everything which needs to be installed or updated first, then update all the tools together. IE Unlike currently, we should be doing only one call to conda install.

(b) The overhead of running things in docker and starting a docker container is pretty low. Take a look at the examples at https://github.com/open-power/pdbg/blob/master/utils/build.sh

(c) We probably want to have a discussion about how configuration works. I'm thinking that we might want to use a kconfig style system but I'm unsure if this would work on Mac or Windows? We can then provide things like "make allconfig" or similar which provide different options?

Do note that litex-buildenv and lxbe are different things.

• LiteX-BuildEnv - https://github.com/timvideos/litex-buildenv
• lxbe-tool - https://github.com/litex-hub/lxbe-tool
• lxbuildenv - https://github.com/im-tomu/foboot/blob/master/hw/lxbuildenv.py

The idea here is to replace + combined all these options into one unified tool which works well for everyone's needs.

[y2kbugger]

Have there been any feasibility experiments for moving to something like yocto?, It seems like a good fit especially for reproducibility.

On Wed, Nov 13, 2019, 17:36 Tim Ansell [email protected] wrote:

There are a couple of things here.

(a) We want to work out everything which needs to be installed or updated first, then update all the tools together. IE Unlike currently, we should be doing only one call to conda install.

(b) The overhead of running things in docker and starting a docker container is pretty low. Take a look at the examples at https://github.com/open-power/pdbg/blob/master/utils/build.sh

(c) We probably want to have a discussion about how configuration works. I'm thinking that we might want to use a kconfig style system but I'm unsure if this would work on Mac or Windows? We can then provide things like "make allconfig" or similar which provide different options?

Do note that litex-buildenv and lxbe are different things.

• LiteX-BuildEnv - https://github.com/timvideos/litex-buildenv
• lxbe-tool - https://github.com/litex-hub/lxbe-tool
• lxbuildenv - https://github.com/im-tomu/foboot/blob/master/hw/lxbuildenv.py

The idea here is to replace + combined all these options into one unified tool which works well for everyone's needs.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/timvideos/litex-buildenv/issues/231?email_source=notifications&email_token=ABORVLPAKJMGZ5JNMEONEZTQTR6PLA5CNFSM4JMY3XHKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOED75FUQ#issuecomment-553636562, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABORVLNP46KC7DGAOEVZFRTQTR6PLANCNFSM4JMY3XHA .

[PiotrZierhoffer]

Just for the record, by lxbe I always meant LiteX-BuildEnv.

[mithro]

Please always use the full name to prevent confusion.

[mithro]

@y2kbugger - We looked at Yocto but it doesn't have good support for building on Windows and Mac (it can build SDKs for Windows and Mac which is a bit different).