Elijah ben Izzy
Elijah ben Izzy
@judahrand yep! Nice callout. Sorry for the delay. I think a big question is what we want to cache... If we cache at the configuration side (E.G. all sync nodes...
We need a build proces... But this is good for now. We're going to get things really out of sync...
Interesting... Another approach is to use some version of pandas that traces the actions and delays it (I bet koalas does something like this, or ibis), then use the intermediate...
This s a WIP, but its on the right track. Things to do: - [x] Simplify the complexity around literal/upstream -- this might not be the best way to do...
Need to run a ray/spark/dask test, but more likely will add some basic integration tests for everything -- E.G> a DAG that uses all our features.
Started looking into numba -- seems like a pretty straightforward thing to test out. Hard to install on my M1, but I'll be giving it a spin with conda soon....
Jax seems to rely on modifying the code -- E.G. importing `numpy` as `jax.numpy` -- which basically decorates/changes types. So not the code-free change we'd want, but potentially useful.
[Numexpr](https://github.com/pydata/numexpr) could be interesting. We could easily compile pandas code to pandas eval, with the following steps: 1. Parse a function into AST elements 1. Validate that they’re numexpr compatible...
OpenLineage looks exciting: https://openlineage.io/.
So, this seems totally reasonable as an approach -- slightly decoupled from circleCI, allowing for easier running of stuff locally (especially outside of a docker image). FWIW, however, using the...