icedb
icedb copied to clipboard
danthegoodman1
An in-process Parquet merge engine for better data warehousing in S3
If someone wants to never delete data (have full point-in-time access), then we can disable the creation of tombstones and never tombstone clean. This means that merges create new files...
Make a directory in this repo called `docs` or `web` and have it be a docusaurus (or similar) app that builds and pushes to something like cloudflare pages. It can...
Pretend to be prometheus and ingest metrics into a table that will allow for easy metric analysis with sql If qryn ends up using IceDB then this wont be needed
Similar to qryn for simple log ingestion from something like cloudflare worker logpush for the datadog api. See https://docs.datadoghq.com/api/latest/logs/ and https://developers.cloudflare.com/logs/get-started/enable-destinations/datadog/ This is how you change it: https://blog.qryn.dev/cloudflare-log-push-with-qryncloud If qryn...
Take in a JSON log, expand to columns, perform queries. We can probably pre-define a data set that is useful and then can run queries against it. Or we can...
Rather than my massive single test file, have distinct unit testing and integration testing. Probably use some framework
If using qyrn and this, need warnings about it creating recursive tracing. We could have it skip certain tables, or have it directly insert rather than go through the tracing...
Ingest directly from segment webhook sink Have 2 tables: One for `event,date` and one for `user_id,date`
Updated and terse example of https://github.com/danthegoodman1/IceDBSegment/tree/main Where combined with either the cloudflare proxy (or this can do it as well) will ingest segment events directly without actually using segment, so...
Metadata
Owner
Metadata
An in-process Parquet merge engine for better data warehousing in S3