UniTAS icon indicating copy to clipboard operation
UniTAS copied to clipboard

Way to test UniTAS properly in unity

Open Eddio0141 opened this issue 2 years ago • 0 comments

In normal unit testing, there's literally no unity runtime. I need some way, preferably automated way (for github actions) to test UniTAS in a similar fashion to unit testing.

This is harder when unity 5 and older versions needs to be tested, since it doesn't have unit testing built in unity. I don't know if this is possible, BUT maybe UniTAS itself could run the unit testing and my project simply needs to supply all unity versions and handle results from each.

Goals

  • Automated testing
  • Built in UniTAS project in some way
  • Able to run on github actions (not sure about this one though)
  • Check all versions from 3.4.0 to latest (not including betas) which BepInEx supports

TODOs

  • [ ] How to handle the test projects sharing similar items across multiple unity projects
    • Of course but it's not going to be efficient for developers to write a script on one test project, then copy across every version like 3.4.0 and 3.4.1 and so on as well as for the scenes to be need to be modified across every version in a tedious way
  • [ ] What versions to test specifically
    • Unity is a bit weird with their changes, but would it really be required to test every single release? For example scripts made in 3.4.0 would most likely work in 3.4.1 without breaking anything, but this might not be the case
  • [ ] Where to get unity versions not on the official archive
    • Funny enough unity doesn't list versions below 5.0.0 on their archive site
    • Also funny enough, you can't download linux versions for many of those archives
  • [ ] What platform(s) to test on
    • Unity's archive page doesn't allow linux clients to be downloaded for every version, only seems consistent with windows binaries. Does this mean linux is out of the question for testing in most part?
    • I'm not sure if BepInEx even works on Mac and other platforms
  • [x] "Game testing"
    • It's not like everyone owns all the games listed on the compatibility list, nor anybody wants to bug those that has or bother getting it themselves
    • Use an "analysis tool" - #286
  • [x] external tool to run tests on unity projects
    • [x] interprocess communication from UniTAS to the tool to the tool to trigger testing
    • [x] running "tests" defined by the external tool, this would be things like running scripts from the external tool and doing certain operations
    • [x] reading internal state of game via repl
    • [x] checking if game crash / unitas crash / coredump / other major things

Note on game testing

I might have to create a tool that analyses a game with the assemblies and assets with scenes, which then reports features to test on unity, which the test projects can implement

  • [ ] #286

Eddio0141 avatar Jul 03 '23 23:07 Eddio0141