vmware
RFC: Reflective API.
Add an API to extract information about input/output relations, and possibly other parts of the program from a running DDlog instance.
@convolvatron, do you have a list of reflective capabilities required by d3log?
@blp, @debnil, do you know what you need for Nerpa?
@Kixiron, this might be another way to expose info about the program to the profiler instead of using regions.
I can imagine two APIs: a static one, where the compiler generates some JSON representation of the relations, and a dynamic one, where you can query dynamically about relations. But I don't know exactly what you can promise in the dynamic API (e.g., current relation size in tuples?)
For Nerpa we switched to a different approach that should not require reflection.
Customer request: report the number of relations, arrangements, rules, etc. It may be useful to do this at compile time. Alternatively, we may add a CLI option to output stats about the program.
I don't think that a static option for this is feasible with our current setup since the timely/ddflow sides of things are pretty much opaque to the compiler. It has things that are similar to the state of the dataflow graph but it doesn't have a 1-1 match for it. For example, an output relation is not a single timely operator, it can consist of all of the following:
- A
.distinct()call (which can require up to 2 arrangements and 3 total timely operators) - A
.consolidate()call (1 arrangement, 2 operators) - A
.probe_with()call (1 operator) - An
.inspect()call (1 operator) ...and so on
So any statistics we try to give from the compiler that make judgements about the dataflow graph's state are going to be long, tedious and inaccurate but I do think that really basic "Number of relations, number of clauses" stats about the written program will work nicely
Yes, that's what I meant, we can potentially output whatever is know to the compiler statically.