jak-project [jak2] Mega Issue for setting up decomp

This is a first pass at all the things we'll need to do before officially starting jak 2's decompilation. I'd really like to make the jak 2 decompilation a little bit more organized and in-order compared to jak 1, so it will be important to get the framework in place.

Before we "start" jak 2, I'd like to have all these completed:

[ ] Documentation of OpenGOAL language
[ ] Decompiler fixes (type pass, stack vars, far labels, stack checking, automatic suggestions for some types)
[ ] Jak 2-specific decompiler tweaks (new loops, new rtype-of, new floats)
[x] all-types for jak 2 auto generated https://github.com/open-goal/jak-project/pull/1614
[x] #1573
[ ] Working goalc for jak2
[x] All decompiler tests (type consistency, decompiler regression, offline) set up and working for KERNEL.CGO
[x] Decompile KERNEL.CGO, kernel tests passing (checks for register corruption), GOAL + C kernel + linker + repl working

I think we shouldn't worry about:

anything related to dma/graphics synchronization - I'd like to do something different for jak 2's variable frame rate
overlord/sound - jak 2's ckernel can load DGOs with jak 1's overlord, which is enough to get started.
[x] #1668
Very specific decompiler stuff, like types of art-dir, that we can't figure out easily yet.

The goal for jak 2 is to be able to have a jak 1 and a jak 2 build at the same time, with as much shared c++ code as possible. Tools will take a command line flag to pick the version and jak 1 outputs/temporary files shouldn't be mixed with jak 2.

[x] remove assets, use decompiler_out for assets https://github.com/open-goal/jak-project/pull/1575
[x] separate "out" folder per game https://github.com/open-goal/jak-project/pull/1585
[x] move fr3's to out/fr3 https://github.com/open-goal/jak-project/pull/1575
[x] move texture output to decompiler_out https://github.com/open-goal/jak-project/pull/1575
[x] remaining functions in C Kernel for jak 2 #1611
[x] support jak2 process stuff in all-type generation #1613
[x] remove duplicate fields in all-types, apply "jak 1 guesses" to all-types #1613
[x] generate all-types2.gc #1614
[x] set up goal_src2 with empty files
[x] Compiler supports multiple projects #1619
[x] game2.gp
[x] compiler takes source directory as an argument
[x] compiler symbol changes
[x] compiler link table changes
[ ] compiler debugger changes
[ ] stack layout checker
[ ] compiler float changes
[ ] overlord jak 2 (I think we can get away with reusing jak 1's overlord if we're just loading dgos)
[ ] GOAL language documentation
[ ] jak 2 decompiler handles new until loop
[ ] new decompiler type pass
[ ] jak 2 decompiler handles new floats (might require above type pass)
[x] jak 2 compiler tests
[ ] jak 2 decomp tests
[x] jak 2 type consistency
[x] jak2 rtype of #1616 (not finished)
[x] jak 2 offline tests
[x] jak 2 kernel decomp regression tests
[x] jak2 KERNEL.CGO decompilation
[x] jak 2 kernel tests
[x] all-types fixup https://github.com/open-goal/jak-project/pull/1728 https://github.com/open-goal/jak-project/pull/1735

Compiler stuff:

Jun 28 '22 01:06 water111

The following seem to be done:

jak 2 type consistency https://github.com/open-goal/jak-project/blob/5148523917795a4b11c8bf5b74e6e1d3df421aaa/test/goalc/test_type_consistency.cpp#L28-L34
jak 2 compiler tests
https://github.com/open-goal/jak-project/blob/master/test/goalc/test_jak2_compiler.cpp
https://github.com/open-goal/jak-project/blob/master/test/goalc/test_goal_kernel2.cpp
jak2 KERNEL.CGO decompilation
- this seems done unless we need to do pskernel or some of the files aren't actually complete / we are unsure of their output?
jak 2 kernel tests
https://github.com/open-goal/jak-project/blob/master/test/goalc/source_templates/jak2/kernel-test.gc

Aug 04 '22 05:08 xTVaser

Yep, agreed on almost all. A few notes:

Biggest missing thing is updated type pass. It's getting close but still some testing/error message/interface with label/stack stuff to do
we need to redo all-types with a few fixes for jak 2 (method names, fix up some issue with location of types with no inspect)
kernel tests and decomp are good
jak 2 compiler tests are a bit... sparse but I'm going to call it good enough. The difference from jak 1 to jak 2 is small and bugs introduced here aren't going to be the hard-to-find type.
compiler float changes: unclear if we actually need to do this - it might be a tiny bit faster to do things the jak 2 way, but no noticeable difference to the code (we already deviate from the game in how we create integer constants, for example. no reason to copy exactly what they did for floats).
For documentation of decompiling, I think a video demo is going to be more helpful than docs. I'll give that a try at some point. We can always go back to docs if that doesn't work out.

My plan for this weekend is to focus on the type pass stuff.

Aug 04 '22 05:08 water111

"can run empty loop in main.gc" I think we can drop this too. I tested already that we can do empty loops in a process.

Aug 04 '22 05:08 water111

Sounds good, I'm working on jak 2 kernel decomp regression tests because it seems like we only had ones for jak 1 / cleaning up the text fixtures for it.

But documentation sounds like a good next step for me as well -- now that I've finished off most of my immediate plans for the tooling around decompiling, I'll try to throw together some docs on how to use them.

Agreed that a video is a good idea, I think in general it will be difficult to update it (ie. we find a new scenario we'd like to document as an example). Much easier to add that with images/text. But for the people that prefer videos, having atleast one as an overview / introduction is a good idea.

Aug 04 '22 05:08 xTVaser