apache
[Bug report] list-table api is very slow when table quantity is very large
Version
main branch
Describe what's wrong
Through my test,I found that list-table will takes 300s when a schema has 5000 tables .
I analysis the code and add some logs ，then found is the reason for calling the getTableObjectsByName interface.
listtable use the getTableObjectsByName .this metatore interface is very slow.
Error message and/or stacktrace
I add some logs at 3 positions.
the result is:
How to reproduce
add 5000 tables to one schema
Additional context
No response
I found that using this getTableObjectsByName mainly to filter out inner and outer tables, as well as to filter out iceberg tables. If I don't filter the inner and outer surfaces. What is the impact here? What additional types of tables will be return?
Can you provide the direct query time for HMS without Gravitino?
we have tested id。 when I excute "show tables" in hive beeline and sprark .it is very fast. I gusess hiveserver2 don't use this getTableObjectsByName interface .because 'show tables' just return table names.
time1andtime2do not seem to appear in the picture?
sorry, i will send you a new one
I have tryed the listTableNamesByFilter inteface to filter iceberg table。It is a feasible approach. but I did not pay attention to filter the manager and external table，I dont know the point of filtering manager and external table.
so, please check this way. if it is acceptable ,I can submit a pr.
I have tryed the listTableNamesByFilter inteface to filter iceberg table。It is a feasible approach. but I did not pay attention to filter the manager and external table，I dont know the point of filtering manager and external table.
so, please check this way. if it is acceptable ,I can submit a pr.
Great! I think we can work on this way. WDYT? @jerryshao @FANNG1
Great! I think we can work on this way. WDYT? @jerryshao @FANNG1
I think it's ok, because this method seems extensible and not only works for filter Iceberg tables.
