decomp.me icon indicating copy to clipboard operation
decomp.me copied to clipboard

RFC: Project function importing to support transient asm

Open bates64 opened this issue 3 years ago • 2 comments

This is a writeup of an idea for how we can make ProjectFunction importing work for repos that have transient asm (i.e. no asm files in the repo).


The idea here is to move the burden of importing onto the project's existing infrastructure, rather than the decomp.me server. For papermario, this would be Jenkins.

New API endpoint:

POST /projects/papermario/push

{
    "api_key": "[secret]",
    "unmatched_functions": [
        {
            "rom_address": 0x89FEC,
            "create_scratch": {
                "name": "func_800F0B3C",
                "preset": "Paper Mario",
                "target_asm": "...",
                "context": "..."
            }
        }
    ]
}

The project should periodically send a request to this endpoint with all the unmatched functions it knows about, e.g. after pushes to the main branch. Any functions that aren't provided here are assumed to now be matched.

Also, remove ProjectImportConfig and GitHubRepo. We don't need to clone repos anymore.


Alternatively, we might be able to get away with POST /scratch taking a project argument in order to add functions to a project. Removing a function (for example, once it's been matched) would be a DELETE of the scratch. The project then, however, would have to remember to tell us when functions have been matched, else they'll stay on the site, which isn't very fun.

bates64 avatar Sep 18 '22 16:09 bates64

I like this idea in theory, but batch-uploading like this is going to be very hard on the server's disk space, which is already a big problem as-is, not to mention the amount of time this will add to Jenkins builds, uploading over 1000 functions worth of data.

I still think we should do this, but the process should only be responsible for uploading metadata, rather than full data. If someone wants to make a scratch with a transient asm repo, I think what we should do is provide a good cli tool that uses a configuration file and allows people to make scratches on-demand. When someone goes to attempt a project function, it could tell them exactly what to run, for example.

It's not perfect, but I really don't want to be uploading entire projects worth of data to the site at once, especially as part of ci builds

ethteck avatar Sep 18 '22 17:09 ethteck

Some notes

  • Make projectfunction just a scratch owned by a project, pushing, updating etc funcs should just be normal scratch api calls
  • Link to project from scratches
  • PR from scratch page

bates64 avatar Sep 26 '22 01:09 bates64