Format for running tests

[JimLewis]

I noted in your pull request for OSVVM that you run tests by saying:

      - name: Test Rand
        run: orbit test --tb Demo_Rand --target gsim -- --std=08

Let me point out a couple of things:

1. this is quite verbose for running just one thing.
2. you have mixed your tool to run with a test case. You need to separate the simulator from the test case as this is awkward if you need to test against multiple simulators.
3. running a set of tests often requires additional files that must be compiled before this.
4. VHDL supports libraries. To keep different test harnesses separate, I recommend using libraries.
5. OSVVM requires conditionals based on tool, tool version, variable settings.
6. What do you do for test case artifact management - creating a simulator transcript, managing test output files? In particular some methodologies such as OSVVM produce yaml output files that get translated to HTML by the scripting.
7. how do you manage two test cases producing test output files that have the same name?
8. How do you support conditionals? OSVVM requires conditionals to support different tools, tool versions, and settings.
9. How do you handle grouping tests into test suites?
10. is building the design and/or running the test cases in any way hierarchical?

You can certainly solve some of this in YAML or other format, however, tests create output. Managing and post processing these may be methodology specific and likely require the methodologies runner.

Even building RTL designs is going to eventually require some sort of conditional support for variations of clock managers for the different FPGAs.

[JimLewis]

Going further, for projects that use a variety of tools, the value that you can bring is:

1. Identify a repository version
2. Identify how to get it
3. Identify how to build it.

This would help projects that bring different methodologies together that do not have the same build methodology.

For simple projects without their own build manager (ones just calling a single tool) it may be worthy of having a simple build process

[chaseruskin]

Thanks for looking into the project and providing some feedback. I'll try to address all your comments here:

1. Options for setting default target and letting Orbit automatically detect top-level can let the user not have to explicitly set the options (--tb and --target). All options after -- are sent to the target workflow (I used GHDL for that example). For commands that a user has to supply many options, there can always be additional layers of abstraction such as writing a simple makefile recipe to run this given command. I was overly verbose in the CI/CD so it can be well understand what options were ran when looking at logs.
2. The tool is separate from the simulator. The tool does all the heavy-lifting of finding what files are needed for a simulation, and then outputs a file that lists all the required files. It's on the target to interpret this file and run whatever EDA tool you choose. For this example I made a script called gsim to process that file list and run my GHDL simulation. With this architecture, any simulator/EDA tool can easily be configured to work with Orbit.
3. Targets can tell Orbit what additional files you need (such as PDC and SDC files) and can include them in the file list for the target process to handle them in whatever order they want to.
4. Orbit supports defining your library for an IP in its manifest. This library is reflected in the file list and can be used when processing the file list with whatever simulator you are working with.
5. You write your own script to wrap whatever tool/version and options you always would require, so the user has complete control over letting their EDA tool be configured correctly based on what they have.
6. Artifacts of any build/test are stored in the target output directory of an IP (typically called a directory called build/). Since the user is the one who wrote the wrapper script for their target EDA tool, they have control over what files to create and keep after a run.
7. Not too sure I understand this one. Again, you could handle this in the user-level target wrapper script, but every time you run a build/test command Orbit cleans the "build/" directory (unless using --no-clean option), so your test is always clean and contains the immediate results.
8. See 5.
9. Again, if you want to run a group of tests, you can either just always run these commands, or explicitly automate the test suite into the target wrapper script that goes around your EDA tool.
10. Yes, the file list Orbit produces that must be processed by your wrapper script generates the list in topological circuit hierarchy order, using only the dependent files the specified top-level requires, so a user will never have to manually write a file list for their EDA tools.

Again, writing a wrapper script around your EDA tool of choice is up to the user, so you can always add as much complexity as needed, such as wanting to also process YAMLs that contain a test suite that you want to run on the simulator. Possibilities are quite endless in how you design your workflow automation between Orbit and your EDA tools.

Thanks again for providing your feedback, and hopefully some of these responses clear up any confusion. I hope to continue to better document the tool so it is easier to understand and continue to refine its system. Thanks @JimLewis !