Relocate files/dirs

[klesh]

What and why to refactor

Many folders were randomly located in the wrong folder, and sometimes, they caused cycle import error, and the root folder is bloated

Describe the solution you'd like

• #2156
• remove scripts folder
• move CONTRIBUTING.md to lake-website
• move plugins/*.md to lake-website
• move k8s-deploy.yml to releases and update documentation
• rename plugin_db_migration.go to migratable
• remove e2e folder
• remove plugins/helper/common/callbacks.go

To reorganize the packages/folders to the following structure.

Document

• Add the new structure image and description to Developer Document

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.

[github-actions[bot]]

This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.

[keon94]

bump

[keon94]

@klesh

1. Do you have any suggestions for where any framework-level python code should go (from #3701)? Should we create a top level directory called "python" and throw all *.py files in there? Or should we place them next to the *.go files at the appropriate places? (I personally prefer the first option because it keeps things well-isolated)
2. I've used the scripts folder for the model generator of Singer taps (#3123). Do we still want to remove it?

cc @CamilleTeruel

[klesh]

@klesh

1. Do you have any suggestions for where any framework-level python code should go (from [Feature][Python plugin SDK] Support plugin development in Python #3701)? Should we create a top level directory called "python" and throw all *.py files in there? Or should we place them next to the *.go files at the appropriate places? (I personally prefer the first option because it keeps things well-isolated)
2. I've used the scripts folder for the model generator of Singer taps ([Feature][Framework][POC] Streamed data collection using Singer Spec #3123). Do we still want to remove it?

cc @CamilleTeruel

@keon94

1. I think the top-level directory makes sense. Everything related to python-support should be put into it including *.go files if they are meant for python plugin support
2. I don't have an opinion on this, it is up to you to judge.

[keon94]

@klesh I was thinking, how about we put everything go-related under a broader "go" directory as well? Since config-ui contains all frontend related logic, I think a dedicated folder for backend will also help with the organization. It also makes more since as we are using a top-level python/ directory as well in the Python SDK branch. So something like go-backend/ which contains all the modules/packages, the go.mod/sum etc. Maybe py-backend in that other branch for python stuff.

[klesh]

@klesh I was thinking, how about we put everything go-related under a broader "go" directory as well? Since config-ui contains all frontend related logic, I think a dedicated folder for backend will also help with the organization. It also makes more since as we are using a top-level python/ directory as well in the Python SDK branch. So something like go-backend/ which contains all the modules/packages, the go.mod/sum etc. Maybe py-backend in that other branch for python stuff.

Good thinking, I would like to share my 2cents as well:

1. both go/python belongs to the Backend, while go acting as the dominating role controlling the whole operation and python serving as a optional helper for plugins, it makes more sense to put python inside the folder containing our go code.
2. should we keep the backend/frontend code inside the same repo? there were discussion talking about extracting config-ui to another repo.

It is big, let's do it step by step, finish the go module relocation first.

[keon94]

Good thinking, I would like to share my 2cents as well:

1. both go/python belongs to the Backend, while go acting as the dominating role controlling the whole operation and python serving as a optional helper for plugins, it makes more sense to put python inside the folder containing our go code.
2. should we keep the backend/frontend code inside the same repo? there were discussion talking about extracting config-ui to another repo.

It is big, let's do it step by step, finish the go module relocation first.

1. Yeah, it's just kind of awkward to mix go.* files with, say, requirements.txt or *.toml in the same top directory. If in the future we decide to add a third language as well it'll just get messier. How about backend/go/* and backend/python/? backend-go/* and backend-py/ would achieve the same thing as well. I guess it's a matter of preference. I can also settle for having python coexist with Go under the same backend/ directory - not a huge deal to me.
2. I agree, we should do that in another PR if we decide that.

[hezyin]

As you guys mentioned, it's really a matter of preference. Personally, I'd vote for having both backend/go/* and backend/python/* and there's a plugin folder under each language folder (e.g., backend/go/plugins and backend/python/plugins).

[keon94]

Another solution, to keep the directory levels fewer, is to just move everything Go to backend/. And the future Python code to backend-py/. Having a simple "backend" directory without a suffix would imply that this is the main backend directory (Go).

Edit: On second thoughts, backend/go and backend/python will be better, because this way there'll be a natural way of placing shared resources; i.e. Makefile, Dockerfile, scripts, etc, would simply go inside backend/ and be common to both.

[keon94]

Here's what I have at the moment:

[klesh]

I would prefer to keep backend for all backend code including go/python and move all python into backend/python folder including requirements.txt.

[keon94]

Alright, I've moved everything under backend/. Question: there's no interface for Config - we just use Viper directly everywhere, so I can't split it into cfg and vipercfg. I assume this ought to be a separate refactor for later? For now, just keep what we have in the config package (?)

[keon94]

I think we should eventually move to use a dependency injection framework, e.g. https://github.com/uber-go/fx