Make e2e `wasp-app-runner` reusable

[FranjoMindek]

Currently the wasp-app-runner is located in the root of the wasp repository.

This makes it hard to reuse outside of wasp repository, and makes it hard for wasp users to use it as well.

The base case scenario would be to make it a npm package, so it's easily available both to users and us (e.g. in CLI or locally). But currently we holding out of npm.

Second best scenario is to make it a Github action. But to do that we must move it to a separate repo, as actions can only be defined in the root of the repository. The additional bonus is that the wasp-app-runner is easier to clone now before we sort out the npm situation.

[Martinsos]

Why are we holding out of npm right now?

I also didn't get this one:

The additional bonus is that the wasp-app-runner is easier to clone now before we sort out the npm situation.

[FranjoMindek]

Why are we holding out of npm right now?

Do we have our organization name sorted on npm? From what I know, when we brainstormed about it, we didn't want to release the npm packages yet.

I also didn't get this one:

The additional bonus is that the wasp-app-runner is easier to clone now before we sort out the npm situation.

It's not deeply nested in wasp repo, but in the root of runner-action. Nothing much more.

[Martinsos]

Do we have our organization name sorted on npm? From what I know, when we brainstormed about it, we didn't want to release the npm packages yet.

I think not, but we will solve that if the need comes for it, that shouldn't be stopping us, look at it as an implementation detail.

[FranjoMindek]

Since we never properly explained our thinking, (we did have a on-call discussion, but never wrote the results and further expanded on them) I'll try to summarize the current situation.

Backstory

My initial take on this task was posted on discord, so let's build on it:

We now saw 4 total approaches to this problem:

1. Implement wasp-app-runner as a Github action
2. Publish wasp-app-runner as a npm package
3. Integrate it into wasp-cli as a command
4. Integrate wasp-app-runner as part of wasp repo, but not wasp-cli

The purpose of wasp-app-runner is to make it easy to run headless tests. We are running headless tests: a) in our CI pipeline, in possibly different repositories (e.g. wasp, open-saas) b) locally, even outside of wasp project

Which means that our approach must make it easy both to distribute to the CI pipeline and locally.

1) Implement wasp-app-runner as a Github action

Pros:

• Solves a) trivially

Cons:

• Requires wasp-app-runner to be in a separate repo, due to Github actions constraints
• Doesn't provide an easy way to solve b), we must clone, build and install the package, most likely achieved through wrun script

2) Publish wasp-app-runner as a npm package

Pros:

• Monorepo
• Solves a) and b) trivially, wasp-app-runner is part of devDependencies, easily globally installed too
• Easily used by non-Wasp team members

Cons:

• Maintaining an npm package

3) Integrate it into wasp-cli as a command

Pros:

• Monorepo
• Solves a) and b) trivially as we ship it together with wasp-cli

Cons:

• We ship it with wasp-cli. Is it really a command we want to have? Is running headless tests part of wasp-cli concerns? Most likely not.
  • Additionally it clashes with another command currently being implemented: wasp build start. What's the difference between those two. Would be complicated to explain everything properly.

4) Integrate wasp-app-runner as part of wasp repo, but not wasp-cli

Pros:

• Monorepo

Cons:

• Hard to do a) in non-wasp repos
• Would depend on wrun script for b)

Direction

Our starting position was approach 4), which allowed us to easily add headless tests to todoApp, and leaned on wrun to make it accessible. But we wanted it be more reusable across different repos, so 1) 2) and 3) remain.

During the discord discussion we decided that 3) was not something we wanted to do. It really wasn't a command we wanted to have as a part of wasp-cli, so we removed that option.

During our on-call discussion, we decided that we didn't have our npm organization situation resolved, so we decided to abandon the npm package idea, at least until we sort it out.

Finally, because of above we settled on 1), which was the direction that #2807 was going for.

I think not, but we will solve that if the need comes for it, that shouldn't be stopping us, look at it as an implementation detail.

If we really don't mind the npm organization issues we currently have, it may be best to pivot the current Github action approach to the npm package approach.

@Martinsos @sodic @infomiho @cprecioso

[Martinsos]

Ok, nice analysis!

I agree it sounds like it shouldn't be part of Wasp CLI (option 3).

And we shouldn't let npm org thing stop us, we will just create / use org we need.

Option 1 doesn't sound great to me, having it as GA.

So based on this analysis, let's go with npm (option 2).

One thing that is not clear to me is, what is our npm release process going to look like for this? How will it complicate our overall release process?

Maaaybe also option 4 might make sense btw. I guess we need a bit of extra tooling or something to run it in some scenarios, but I imagine it would be quite simple. And we don't have to deal with the npm.

I think I would be interested in undresatnding the option 2 vs option 4, in the sense of "pain of releaseing on npm" vs "pain of running it if it lives in wasp repo", so I can undersatnd which is worse for a) and b) situations and for us.

[infomiho]

Great analysis Franjo!

My hunch is. that if have the wasp-app-runner as a npm package will make it easier for us and our users to use. It's a tool that we distribute as an npm package that it's a simple and familiar way to install global tools. Publishing it as an npm package can be done IMHO manually at first to keep it as simple as possible. The code won't change that often.

I'd avoid pulling the runner in Wasp core as it's just an orchestration tool for running Wasp and Docker commands. It doesn't feel like a part of Wasp CLI (this reminds of Unix philosophy and focused tools). We did start to move some of the logic from the wasp-app-runner into wasp build start which would enable us to just invoke wasp build start from the runner - reducing it even more to a wrapper for calling Wasp commands.

Immediate future: I'd probably keep using the wasp-app-runner as-is (not rush the GIthub Action change the we have in a PR) to add testing for starters first to keep us moving forward and not block on wasp-app-runner.

[sodic]

I like the analysis @FranjoMindek!

Some of the options are bundled together.

For example, where the app runner lives in and how we distribute it are related but distinct issues. For example, it's possible to keep it in its own repo and publish it to NPM. Perhaps this way it could also double as a GH action (not sure whether this makes sense, probably not)?

Option 1 doesn't sound great to me, having it as GA.

What don't you specifically like about this one @Martinsos?

I think I would be interested in undresatnding the option 2 vs option 4, in the sense of "pain of releaseing on npm" vs "pain of running it if it lives in wasp repo", so I can undersatnd which is worse for a) and b) situations and for us.

I second this comment from Martin. Although I think I know the answer, it's good to have in writing as well :)

Immediate future: I'd probably keep using the wasp-app-runner as-is (not rush the GIthub Action change the we have in a PR) to add testing for starters first to keep us moving forward and not block on wasp-app-runner.

@infomiho Not sure what you mean here, it sounds like you'd keep the runner in the main repo, but it's already in a different repo "as is."

[infomiho]

@sodic the app runner currently lives in two places and the one in the main repo powers the headless tests still

[Martinsos]

Option 1 doesn't sound great to me, having it as GA.

What don't you specifically like about this one @Martinsos?

The reason I said "sound" is because I don't really know enough about this :D, but it sounded like shaping it into GA makes it harder to reuse outside of GA context, and also it was said in analysis it forces it to be in a separate repo. So sounded quite constraining. That is it.

[FranjoMindek]

For example, where the app runner lives in and how we distribute it are related but distinct issues. For example, it's possible to keep it in its own repo and publish it to NPM. Perhaps this way it could also double as a GH action (not sure whether this makes sense, probably not)?

I'd be best not to introduce multiple solutions if the don't bring any benefit. And I don't see much benefit in using an action instead of npm i -g ... inside of Github workflows. Both are "one liners". Both require node in the end.

Additional info: On discord, @cprecioso also brought up the option of using Github packages: https://discord.com/channels/686873244791210014/1390218158748471376/1390231560296337491 (internal channel link)

I think the biggest benefit of Github packages is centralization and permissions. Since we only have npm, and a public library, I don't see too much benefit for it. Github packages solution would make it less accessible, but do we care about making it "less" public? Most likely no.

One thing I don't like with Github packages is that we would need to have extra setup to make it work wih npm, e.g. tell it where to actually locate the package since its not in the npm registry.

I would stick to the de facto standard, as Github packages doesn't solve any problems.

It seems like we are moving towards 2), publishing the wasp-app-runner as a npm package. It that case I would also like to keep it in the wasp-lang/wasp repo. If so we need to:

1. Update current wasp-app-runner to the latest code from wasp-lang/runner-action, and publish it to npm
2. Update our projects to rely on the wasp-app-runner npm package rather than current approach
3. Archive wasp-lang/runner-action

[sodic]

From what I understand, the NPM approach makes sense.

But before we dive into it, let's make sure we answer some questions from before. Specifically, this one:

One thing that is not clear to me is, what is our npm release process going to look like for this? How will it complicate our overall release process?

Maaaybe also option 4 might make sense btw. I guess we need a bit of extra tooling or something to run it in some scenarios, but I imagine it would be quite simple. And we don't have to deal with the npm.

I think I would be interested in undresatnding the option 2 vs option 4, in the sense of "pain of releaseing on npm" vs "pain of running it if it lives in wasp repo", so I can undersatnd which is worse for a) and b) situations and for us.

Finally, if we're going with NPM, I will add that it might not be necessary to bring it back into wasp-lang/wasp. I think it can stay there a separate thing. I know you listed "Monorepo" as a benefit, but I don't really mind the runner not being in wasp-lang/wasp (and it might play nicer with NPM, not sure).

I suggest picking whichever option is the easiest (which currently seems to be keeping it in wasp-lang/wasp-app-runner. Still, I'm interested to hear the monorepo approach benefits.

[FranjoMindek]

Finally, if we're going with NPM, I will add that it might not be necessary to bring it back into wasp-lang/wasp. I think it can stay there a separate thing. I

Since wasp-app-runner can be in a separate update cycle than Wasp itself I don't see much issue with that. I think that location makes no difference for npm. Only benefits I see are:

1. If we ever need to update wasp-app-runner together with Wasp, it can be done in a single PR
2. Everything is localized together, the point of monorepos

I doubt that 1) is going to happen that much. 2) makes more sense to me. We need to start a new directory for that though, /packages, as waspc/packages doesn't suit this one. If that direction makes sense, we can go that way.

I suggest picking whichever option is the easiest

Honestly, not much difference, if any.

[FranjoMindek]

One thing that is not clear to me is, what is our npm release process going to look like for this? How will it complicate our overall release process?

I never found npm publish process complicated? It shouldn't complicate too much. Especially since it is a separate cycle from Wasp and we won't update wasp-app-runner so often. If its still too long we can make workflow for it.