Tinybird forward compatible - migration guide

[anaguerr]

Opening a new PR following this older one to resume this topic:

This PR contains the necessary changes to make the Tinybird part of Papermark compatible with Tinybird Forward in order to be aligned with Tinybird's newest functionalities. The main goal is to unblock the possibility of testing everything locally, no need to push the resources to the cloud, along with making the process of iterating resources fully managed and transparent.

Description of changes

• Add TYPE endpoint to endpoint pipes. This is the default in Tinybird Classic, and has to be explicit in Tinybird Forward.

• Add an explicit partition key, that was implicit in Classic and reflects the current schema of the data source.

• Remove VERSION from resources, rename the files to include the suffix of the last version, and always refer to resources with the suffix. Since VERSION tag is not compatible with Tinybird FWD; the ideal change would be to remove it and change the name of the datasources in the cloud to remove the suffix, but that would implicate middle steps to migrate the existing workspaces so we consider the proposed path is more straight forward.

• Use baseUrl when defining tb instance to determinate if the user wants to point to the local environment or cloud environment.

• Modified env.example to add TINYBIRD_HOST in order to define baseUrl.

• Unify VERSION of get_page_duration_per_view endpoint: please note that VERSION 4 is defined in the .datasource file but get_page_duration_per_view__v5 is called in pipes.ts.

Steps for new users

For users without existing Tinybird Classic workspaces, the instructions to start using Papermark and Tinybird locally would be: 0. To install and use Tinybird Forward on your machine, you need to have the following prerequisites:

a. A free Tinybird account
b. A container runtime, like Docker or Orbstack
c. Linux or macOS

1. To install Tinybird Forward, run the following command:

curl https://tinybird.co | sh

2. Navigate to the lib/tinybird directory:

   cd lib/tinybird
   

3. Authenticate into your Tinybird workspace:

tb login

4. Start your local environment:

tb local start

5. To deploy the necessary resources (data sources and pipes) locally:

tb deploy

6. [Optional] - To check the deployed resources in the local UI, run the following command:

tb open

7. [Optional] - To deploy the resources to the cloud, run the following command:

tb --cloud deploy

8. Don't forget to set the TINYBIRD_TOKEN with the appropriate rights in your .env file, along with your local host in TINYBIRD_HOST. Your can get your token by running the following command:

tb token ls

Updating Tinybird

To update Tinybird FWD cli, run the following command: 
```shell
tb update

Questions from previous PR

So the explicit partition key on the video_views__v1 is already applied by default, so pushing that update won't break anything on Tinybird, right?

That's right, it won't break anything.

based on your changes, is there anything I should do in the production side of things on Tinybird? So in the future when I add a new pipe, I will add the version in the pipe's filename and then push to Tinybird.

If we merge this changes and you don't want to migrate yet to Tinybird FWD, you don't need to make any change in your production environment. When adding a new endpoint you can decide the name (using the version of not). The only thing that changes a bit is the process when updating an existing resource:

1. Right now you are using the VERSION tag, meaning that you will modify the resource, increase the version from 1 to 2 and then push the same file.
2. If you want to still use Classic with this structure, you will need to duplicate the file, change the name (you will have endpoint__v1 and endpoint__v2) and then push endpoint__v2.

Is Tinybird Forward more than the local container env? The tb --cloud deploy will still deploy to the same Tinybird cloud I'm used to right?

Yes, it would be the same Tinybird cloud but you would need to make the migration described before running a deploy. The workspace will keep the same resources and data but the UI look and the way of interacting with it are a bit different.

Steps to migrate existing workspaces

For users with and existing Classic workspace, the steps would be:

1. Check full compatibility with Tinybird Forward following the steps described in the previous PR. You can skip steps 1 and 4 if you already have a workspace.
2. Once you check that the output of running uvx --from tinybird tb --cloud deploy --check is returning the expected result you need to:

• Open a request to Tinybird support in order to enable the possibility of making an actual deploy to the cloud.
• Tinybird support will activate a FF in your workspace that is going to change the look of the UI a bit. The UI will keep working as if it was Classic but you will see it blocked until you make the deploy.
• You need to modify one little thing in order to be able to run a deploy and complete the migration (if we don't detect any changes the deploy will not actually happen).
• Run uvx --from tinybird tb --cloud deploy to force the migration. From that moment the workspace will be a Forward workspace.

** It is important to note that there is not expected downtime or data lost in this process. ** We are working to ensure that, in the future, users won't need to contact support and will be 100% autonomous in the migration process.

[vercel[bot]]

@Anaguerrer is attempting to deploy a commit to the mftsio Team on Vercel.

A member of the Team first needs to authorize it.

[github-actions[bot]]

Thank you for your submission, we really appreciate it. Like many open-source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution. You can sign the CLA by just posting a Pull Request Comment same as the below format.

---

I have read the CLA Document and I hereby sign the CLA

---

_{You can retrigger this bot by commenting recheck in this Pull Request. Posted by the CLA Assistant Lite bot.}

[anaguerr]

I have read the CLA Document and I hereby sign the CLA

[mfts]

Thanks @anaguerr for the detailed explanation. I'll review it and merge it in :)