docs: mismatch between backend capabilities and docs

[drabastomek]

Speaking mainly about SQLite (but I'd assume this would be true for other backends), there are quite a few methods listed as supported by a specific backend (with an Inherited flag in the docs) but cannot be run and throw various errors.

For a specific example, the documentation lists .sql as a supported method for SQLite but when one tried to run ibis.sqlite.sql(...) or db.sql(...) it throws an AttributeError: 'Backend' object has no attribute '_get_schema_using_query'.

Perhaps this could be added to a CI to test these so (1) backend-functionality matrix could be automatically populated, and (2) certain parts of the docs automatically removed?

[saulpw]

xref: #1992

[saulpw]

Yes, good idea. @gforsyth annotated all the backend tests to xfail if the backend functionality isn't supported, and I think we should do a similar push for the actual methods on each of the backends. We can streamline it into a fairly brute-force effort such that we add to each backend a list of the methods that are notimpl/notyet/never. These functions would then:

• raise a NotImplementedError (or equivalent) for a better error message;
• have no __doc__ string so that mkdocs doesn't include them in the documentation for that backend;
• not be in the class __dir__ so that iPython won't include the method in its tab-completion;
• unfortunately still be in the builtin Python REPL tab-completion (because it manually crawls superclasses).

[gforsyth]

For the example above, we've merged #3617 so that sql no longer appears in the docs for the SQLite backend (and it also raises a NotImplementedError if someone tries it) -- but we do need to figure out (ideally) a non-manual way to track existing functionality.

I'm thinking we might be able to use a coverage report to link up xfails against specific backend methods, but this might be overengineering

[krzysztof-kwitt]

Completing our work on enforcing exception validation should improve the quality of our documentation in this area. Related: https://github.com/ibis-project/ibis/pull/5549

[cpcloud]

I think it's a good idea, but I'm not sure we'll get to automating this any time soon. Closing for now.