test-suite-sql-eval
test-suite-sql-eval copied to clipboard
taoyds
Semantic Evaluation for Text-to-SQL with Distilled Test Suites
I have tried on my macos laptop and a centos server with your classical evaluation code. It returns TypeError: cannot unpack non-iterable coroutine object in the classical_evaluation.py code. The problem...
Hi, The current code compares data frames by trying all permutations of the columns, without using the information about the column names. I guess this is OK for very small...
As you write the data is taken from another repository, but when I test the Geography. Your test set in the pickle file contains 187 test examples that do not...
Thank you for your contribution to give this unified framework to test on different datasets. However, as I noticed in the Sparc dataset, the leader board has 2 metrics. One...
Hi, I was running the evaluation script on my predicted SQL queries on Spider dataset, and I've noticed some for some examples, the evaluation script returns an Exact-Match score of...
Thank you for releasing this, great work! I have noticed that the "classical datasets" in your release always have less examples than in the original release. For example: Academic: 185...
Fix classical evaluation: **RuntimeWarning: coroutine 'exec_on_db' was never awaited** TypeError: cannot unpack non-iterable coroutine object See #9
I download spider dataset from [Spider Leaderboard](https://yale-lily.github.io/spider). When I run the command: `python evaluation.py --db spider/test_database --etype exec --gold spider/test_data/dev_gold.sql --pred spider/test_data/dev_gold.sql` Then returns: easy medium hard extra all count...
select * from car_makers, model_list, car_names, cars_data select country_code from players where country_code not in (select country_code from rankings where ranking > 50); For the above 2 sqls, the evaluation.py...
Currently creating mapping between aliases and actual tables names is performed globally for whole SQL, what is not correct for more complicated once. Lets consider following SQL taken directly from...
