[doit] replace setups/osflow/Makefile with doit task(s)

[umarcor]

This is another PR in the series of subsets of #110 (#162, #163, #164, #165).

setups/osflow/Makefile is replaced with doit tasks, plus some more elaborated Python code. Precisely, subdir tasks contains helper Python code used by the do.py script.

As discussed in #110, the idea is to enhance tasks/project.py in upcoming PRs, for it to be based on the ProjectModel from edaa-org; along with converting other setups/osflow/*.mk sources to Python (see section "Future work" in #110).

[stnolting]

You are really hard-working! ;)

What do you think about keeping all the makefiles at first until the documentation (especially the READMEs) is updated to use pydoit?

[umarcor]

You are really hard-working! ;)

In fact, I did most of these in July :smile:. I'm now mostly reviewing and doing cleanup.

What do you think about keeping all the makefiles at first until the documentation (especially the READMEs) is updated to use pydoit?

For now, I converted this PR to a draft, and I will do the enhancements to the documentation before considering merging it.

Furthermore, while I do understand why some users might want to keep the Makefile entrypoint, honestly, I have mixed feelings about it.

The main purpose of using doit is having a single entrypoint for users to see what they can execute (./do.py list --all). Hence, the sooner we get people used to doit and ./do.py Example -b <BOARD> -d <DESIGN> <TASKS>, the easier it will be to modify "internals" without them noticing. Precisely, the next step I'd like to address is converting filesets.mk into Python. If users keep relying on the Makefile entrypoint, we cannot do that, because that'd depend on filesets.mk. On the other hand, if we preserved all the .mk files despite including the functionality in Python/doit, we'd be effectively duplicating the maintenance effort, because the filesets in python would need to be kept in sync with the makefiles.

The only motivation I see for sticking to makefiles is the inability to install Python and/or doit on some isolated environment. For those use cases, I'd prefer if we supported generating an ad-hoc makefile or script. For instance:

./do.py Example --out-makefile=BOARD_DESIGN.makefile -b <BOARD> -d <DESIGN> <TASKS>

would generate a simple makefile which includes the data/variables for a single board and a single design. This is similar to GHDL's feature for generating a makefile, after the sources of a design are analysed/imported. That would allow to generate makefiles "offline" and then run it on isolated environments.

Moreover, I'm concerned with extending the capabilities of setups/osflow. The current makefile/.mk based solution does work, but it is convoluted and not very idiomatic. Wrapping NEORV32 as an IP core in Vivado/Quartus is possible, but quite manual. In order to improve the interaction with different toolchains, and for allowing users to more easily extend the Processor/SoC, using makefiles is a stopper.

Overall, although we don't need to solve/decide it in this PR, I believe it is important that we discuss if we'll need to preserve the makefile approach, or if the proposed makefile generation alternative is acceptable.

/cc @tmeissner

[stnolting]

I totally agree and I am really looking forward to have a unified execution system. All those makefiles are (annoying) to maintain and document... I am just worried that we break existing setups the user have on their local systems. On the other hand, we are doing that every time there is non-backward compatible change in the actual hardware... :thinking:

I just found myself in backwards-compatibility-hell after the major change of the low-level hardware access mechanism (#158). 😅 That is the reason for the new neorv32_legacy.h file. I don't like having this file at all - I have converted all my software but I cannot expect users, who are doing long-term projects with this core (which some people are actually doing! :+1:), to do the same.

./do.py Example --out-makefile=BOARD_DESIGN.makefile -b <BOARD> -d <DESIGN> <TASKS>

I think this would be a nice feature for the aforementioned situation :+1:

For now, I converted this PR to a draft, and I will do the enhancements to the documentation before considering merging it.

Thank you so much! :heart:

[stnolting]

@umarcor Since we have neorv32-setups now this seems to be outdated and we can close this?! 🤔