dataprep
dataprep copied to clipboard
DataConnector should throw correct error if universal parameters are used but not defined
Describe the bug
Currently, when there are no definitions for the universal parameters but is used in the query
, it throws an assertion error instead of something more informative.
To Reproduce For example, this piece of code gives you an error.
dc.query("videos", q="covid", part="snippet", _count=100)
which is because, for youtube/videos, we don't have pagination implemented. But it gives you an assertion error.
Expected behavior Clearer error message.
Looks like this issue is now already taken care of in connector.py
:
Although, the behavior is not consistent across all universal parameters:
-
When
_count
parameter is used and pagination is specified, it is ignored with a warning: -
When _q is used when
search
is not specified, an exception with an appropriate error message is thrown:
@dovahcrow and @peiwangdb, this looks like the right thing to do. Please let me know if this looks okay or if anything needs to be further worked upon.
Below are my thoughts on handling those errors. First, for both cases, in the error description, I wonder if we should tell the user explicitly that the universal parameters should be defined in the config file. For the _q case, throwing an error looks good since there is no other way to proceed. For _count, the possible cases:
- no _count definition, (works the above way, and _count is ignored)
- _count is smaller than a single page result number (r), (no problem)
- _count works properly, between r and r*max_page (no problem)
- _count is larger than r*max_page. (only the retrieved results will be returned. should we also add a prompt to let the user know the number of returned results is smaller than expected?)
@pallavibharadwaj could you double-check the code logic to make sure the execution is aligned with our expectation? And test and think about if there are other error cases we missed for those two parameters.
Thanks!