papermark icon indicating copy to clipboard operation
papermark copied to clipboard

Tinybird forward compatible

Open eclbg opened this issue 11 months ago • 2 comments

Description of changes

Changes to make the project compatible with Tinybird Forward:

  • Add TYPE endpoint to endpoint pipes. This is the default in Tinybird Classic, and has to be explicit in Tinybird Forward.
  • Add an explicit partition key, that was implicit in Classic and reflects the current schema of the data source.
  • Remove VERSION from resources, rename the files to include the suffix of the last version, and always refer to resources with the suffix.

The first and second changes are completely inoffensive.

The third change deserves some comments: pipes will be updated atomically for Tinybird Classic with the usual tb push -f, but data sources will refuse to update. This is not an issue because the changes in files in this PR don't actually change the resources in Tinybird. However, removing VERSION and explicitly pinning the resources to the latest one will make iterating them with Tinybird Classic more cumbersome, if anyone does this.

Happy to discuss these changes in this PR, or you can email me at eclbg at tinybird.co if you prefer.

How to check full compatibility with Tinybird Forward

These steps rely on uv (installation)for working with the Classic and Forward CLIs without issues.

  1. Create a new Tinybird Classic workspace.
  2. In main branch of the repo
  3. Authenticate with Classic CLI: uvx --from tinybird-cli tb auth --interactive. Select region and use admin email token.
  4. Push files to workspace: uvx --from tinybird-cli tb push lib/tinybird/
  5. Use the Forward CLI to check that the first Forward deployment detects no changes: uvx --from tinybird tb --cloud deploy --check. Output should be:
Running against Tinybird Cloud: Workspace <your workspace>

» Validating deployment...

* No changes to be deployed
* No changes in tokens to be deployed

eclbg avatar Apr 07 '25 14:04 eclbg

@eclbg is attempting to deploy a commit to the mftsio Team on Vercel.

A member of the Team first needs to authorize it.

vercel[bot] avatar Apr 07 '25 14:04 vercel[bot]

@eclbg Thanks for the PR 🙌

Add an explicit partition key, that was implicit in Classic and reflects the current schema of the data source.

So the explicit partition key on the video_views__v1 is already applied by default, so pushing that update won't break anything on Tinybird, right?

However, removing VERSION and explicitly pinning the resources to the latest one will make iterating them with Tinybird Classic more cumbersome, if anyone does this.

based on your changes, is there anything I should do in the production side of things on Tinybird?


So in the future when I add a new pipe, I will add the version in the pipe's filename and then push to Tinybird.


A question to Tinybird Forward: Is Tinybird Forward more than the local container env? The tb --cloud deploy will still deploy to the same Tinybird cloud I'm used to right?

mfts avatar Apr 07 '25 17:04 mfts