leoluk

Make TTL configurable via config and reconcile it

1

We currently default to 1 week and tell users to customize it using `alter table default.flows_raw modify ttl Date + toIntervalDay(20)`. https://github.com/monogon-dev/NetMeta/blob/2bab7d91da7c8fd97e8a414989f42c8b8dfd2b69/deploy/single-node/schema/0001_create_flows_raw.sql#L79 Now that we auto-reconcile, we should make this...

Evaluate using buf's protoc to get rid of C deps

4

Much of the work done by Bazel during a clean NetMeta build is related to building protoc. Perhaps we can use buf's pure-Go protoc instead? Probably a performance win, too.

Collect AS paths via BMP

1

-> embed https://github.com/bio-routing/bio-rd Slightly tricky - might be hard to do via dictionary lookup, better to have a separate Kafka topic for postprocessing.

Carefully consider flows_raw sorting key

1

- Optimize for main dasboard queries. - Low-cardinality values first for easy filtering (verify that this does what we think it does). - Sequence Num for sampling, but how? It's...

Use mutation cronjob for applying TTLs

2

The ClickHouse table TTL mechanism is very static - a delete_date is set whenever a part is created and won't be updated when the TTL is altered. We should use...