dlt
dlt copied to clipboard
dlt-hub
Added deploy with modal.
Deploy Preview for dlt-hub-docs ready!
| Name | Link |
|---|---|
| Latest commit | c97e935ca114a66ab3986cb9911ffe84a09df487 |
| Latest deploy log | https://app.netlify.com/sites/dlt-hub-docs/deploys/672b9b743cba580008a2eca2 |
| Deploy Preview | https://deploy-preview-1805--dlt-hub-docs.netlify.app |
| Preview on mobile | Toggle QR Code...Use your smartphone camera to open QR code link. |
To edit notification comments on pull requests, go to your Netlify site configuration.
![netlify[bot] avatar](https://avatars.githubusercontent.com/in/13473?v=4 "Published by a pub.dev verified publisher"){kind=small}
the more i think about it actually, maybe it makes sense to write a really pared down example for this space that is runnable end-to-end for the user (e.g. using duckdb) and linking out to our blog post for a "real-world example". happy to help contribute a pared down example
here's a simpler gist that should just work if you run modal run dlt_example.py and will deploy a daily scheduled job with modal deploy dlt_example.py.
i think this section will have better engagement if the user can simply copy-paste a script and it works immediately; we can adapt this to your docs style and perhaps just link out to the original blog post as a more detailed, real-world example of dlt (i also need to update that one to be compatible with 1.1.0).
lmk what you think! also happy to chat i know ive shared a lot of info here haha.
https://gist.github.com/kning/6a2af9e08ebaad0e486968f98c1939be
@kning hey! Thanks for your thoughts here, your idea with testing is great! We actually practice this, you can find here some getting started snippets that we test on every CI/CD run. We can also add your gist to our testing process, we just need to understand what we call a successful run, we can run this command modal run dlt_example.py on every CI/CD run and stop it immediately if it ran without errors, is that enough? or should it be deployed as well?
running modal run dlt_example.py should be sufficient, but you'd also need to set up a Modal account and set the MODAL_TOKEN_ID and MODAL_TOKEN_SECRET variables in your CI environment.
also checked out the snippets, are those ever surfaced in the docs? i guess i'd expect it to be synced with the snippets on this page but it looks different.
@kning
MODAL_TOKEN_ID and MODAL_TOKEN_SECRET variables in your CI environment.
It shouldn't be a problem.
i guess i'd expect it to be synced with the snippets on this page but it looks different.
We changed our docs significantly recently, and getting started page was removed and replaced with intro. Relevant example you can find here: doc and snippets
i see. how do you think we should move forward then with the modal snippet? ideally i'd like to see a "deploy with modal" page that explains how to create a modal account and the runnable code snippet (which should also regularly be run somehow to ensure that it's correct) and finally a link to the blog post for a "real-world" example. but i guess from what i understand it seems that code in the docs page and CI/CD snippets are managed separately?
@dat-a-man will do that, he will create a snippet file with example and use tags to ingest this snippet into doc page, here we will run modal command to test it
left a few tiny comments, but any other hold up here? would love to get this published soon!
@kning hey Kenny! Everything is ready to be merged, we just need modal token for tests, is it possible to run it without token? does this token expire? how often we should refresh token? can we do that automatically? Can you help us with it?
In current implementation we only run snippet as a python script via pytest, I added simple test inspired by this example
you need to create a modal account / token for yourself, we actually have a guide for this: https://modal.com/docs/guide/continuous-deployment. fwiw, i wouldn't consider that blocking if it's too much work for now i know you've been working hard on this for a long time and i can always help you figure out the ci / cd modal part later.
@kning let's move this conversation in our slack :D I contact you there, I think it will help us to solve this problem faster
woohoo! i don't see it on the docs website yet though is there a separate deploy process?
@kning You can see it in devel version here (it will be publish in main documentation with next release) https://dlthub.com/docs/devel/walkthroughs/deploy-a-pipeline/deploy-with-modal
great! reading this there are some parts of the modal API (e.g. proxy, etc.) that have updated, i can go ahead and open a pr later.