ddlog
ddlog copied to clipboard
HazyResearch
Difficult to manage multiple extractors in one ddlog file
Our ddlog file currently has more than 1000 lines of code (without comments) and we are still planning to add more extractors and factors.
We need modules here. Not only the ability to run different pipelines separately (as described in #50), but also the ability to put ddlog code into different files and then include these files in a main file.
This seems reasonable, but it may take some doing. @netj ... we probably need to think a bit about this.
Running part of a ddlog program (certain pipelines or relations) is important and both ddlog and deepdive needs to be changed to support the right granularity. Can you share some of your use cases on how you run or would like to run the huge ddlog program piecewise?
On the other hand, #46 is related to splitting up large ddlog code. By using the familiar #include directives and cpp (C PreProcessor) command, you can code in multiple files without fighting with ddlog compiler itself. The file name and line numbers in the potential error messages will become out of sync, but it can be a pragmatic workaround for the moment (and a good motivation for ddlog's supporting the cpp source file/line directives).
If you have some shared declarations in shared.ddlog, and have several applications:
- app1.ddlog
- app2.ddlog
- ...
that look like:
#include "shared.ddlog"
# app specific rules
Q?(...).
Q(...) :- ... .
Then you can preprocess app.ddlog right before running deepdive:
cpp app1.ddlog >app.ddlog && deepdive run
Personally I think relying on the good old, extremely robust cpp has so many advantages over creating our own, unless we want to provide a well-designed sophisticated namespace/module system or prevent people from abusing C-style macros in ddlog. It'd be great to know more concretely about any specific things your large codebase demands from a module system that's smarter than just concatenating files.
AGREED!
On Sat, Sep 26, 2015 at 8:53 AM Jaeho Shin [email protected] wrote:
Running part of a ddlog program (certain pipelines or relations) is important and both ddlog and deepdive needs to be changed to support the right granularity. Can you share some of your use cases on how you run or would like to run the huge ddlog program piecewise?
On the other hand, #46 https://github.com/HazyResearch/ddlog/issues/46 is related to splitting up large ddlog code. By using the familiar #include directives and cpp (C PreProcessor) command, you can code in multiple files without fighting with ddlog compiler itself. The file name and line numbers in the potential error messages will become out of sync, but it can be a pragmatic workaround for the moment (and a good motivation for ddlog's supporting the cpp source file/line directives).
If you have some shared declarations in shared.ddlog, and have several applications:
- app1.ddlog
- app2.ddlog
- ...
that look like:
#include "shared.ddlog"
app specific rules
Q?(...). Q(...) :- ... .
Then you can preprocess app.ddlog right before running deepdive:
cpp app1.ddlog >app.ddlog && deepdive run
Personally I think relying on the good old, extremely robust cpp has so many advantages over creating our own, unless we want to provide a well-designed sophisticated namespace/module system or prevent people from abusing C-style macros in ddlog. It'd be great to know more concretely about any specific things your large codebase demands from a module system that's smarter than just concatenating files.
— Reply to this email directly or view it on GitHub https://github.com/HazyResearch/ddlog/issues/57#issuecomment-143465476.