fix: log DDL SQL in verbose mode
Before (and still), we log the SQL inside Backend.compile(). I think what this means is that we are logging all SELECT statements. But, any DDL statements like from create_table() were not logged. Now they are. Some SQL statements may be logged twice now, in .compile() and in .raw_sql(), but I don't think that's a big problem?
Logging twice is not great. Production systems tend to already have problems with log spam, and we should not add to that problem.
OK, that makes sense. If I come up with a way to not duplicate the logs, is this good to merge?
I have several times run into bugs that are surfaced when I .cache() a table (and therefore .create_table() is called under the hood), and it is currently quite annoying to debug what SQL is actually getting executed. Setting ibis.verbose = True and getting a good first idea would be very pleasant.
@NickCrews Can you talk about what problem is being solved here? Logging may or may not be the solution, but I'd like to get some more information about your needs/use case beyond "debugging SQL queries produced by Ibis" if possible!
- the rare use is when there is a bug in ibis and the DDL errors, and then as a maintainer I have to go add print statements to see what went wrong, when I'd like to just see the logs.
- as an end user, I have at this point a very complex library, with .cache calls sprinkled throughout it. I want to improve the performance, so I want to be able to coordinate what bits of my code are causing really big or really slow SQL queries. If ibis logs, then those logs are going to get interspersed with my application logs, and hopefully I'll get a sense of where the slow executes are happening.
Actually, this gives me the idea that it would be awesome if ibis actually timed how long each DDL query took and logged that. Maybe a different log level or something.
Is that enough background info?
I also have been playing with the idea of separating out the DDL compilation from execution. Then, it would be possible to do something like rich, to premake scripts that can be executed in environments without python/ibis installed.
with backend.capture() as capture:
backend.create_table(...)
my_sql_path.write_text(capture.get())
Can file as a separate issue but this might be relevant for this discussion.