presto icon indicating copy to clipboard operation
presto copied to clipboard

Procedure to invalidate directory list cache

Open agrawalreetika opened this issue 1 year ago • 5 comments

Test plan - (Please fill in how you tested your changes)

Please make sure your submission complies with our Development, Formatting, and Commit Message guidelines. Don't forget to follow our attribution guidelines for any code copied from other projects.

Fill in the release notes towards the bottom of the PR description. See Release Notes Guidelines for details.

== RELEASE NOTES ==


Hive Changes
* Introduce system procedure to invalidate directory list cache in Hive Catalog

agrawalreetika avatar Jun 08 '23 06:06 agrawalreetika

@agrawalreetika Instead of single directory, we may want to clear the whole cache too. May be make the directoryPath optional, and if its not provided, then clear all cache. Also, please change the procedure name to something more intuitive.

If there are multiple hive catalogs, how can we invalidate the cache from one particular catalog?

system.invalidate_file_list_cache(Optional<String> directoryPath)

nmahadevuni avatar Jun 09 '23 05:06 nmahadevuni

@agrawalreetika Instead of single directory, we may want to clear the whole cache too. May be make the directoryPath optional, and if its not provided, then clear all cache. Also, please change the procedure name to something more intuitive.

If there are multiple hive catalogs, how can we invalidate the cache from one particular catalog?

system.invalidate_file_list_cache(Optional<String> directoryPath)

Hi @nmahadevuni, Thanks for your comment. I have added the directory path as optional to allow full directory list cache invalidation. Just like any other procedure in Presto we will have to provide catalog name while making procedure call on specific catalog. (ref - https://prestodb.io/docs/current/connector/hive.html#procedures)

agrawalreetika avatar Jun 12 '23 05:06 agrawalreetika

@tdcmeehan Could you please help me review this?

agrawalreetika avatar Jun 14 '23 14:06 agrawalreetika

Codenotify: Notifying subscribers in CODENOTIFY files for diff 908327e1e99bdf4728aa6cbf9f7eb5d5c94d7737...004ba4bf4ceea9f0dc9b39293cb281fd958af13f.

Notify File(s)
@steveburnett presto-docs/src/main/sphinx/connector/hive.rst

github-actions[bot] avatar Dec 01 '23 12:12 github-actions[bot]

Failing on one test (test / test (:presto-hive) (pull_request), and no option to re-run failed tests.

steveburnett avatar Jan 29 '24 14:01 steveburnett

@tdcmeehan I have been facing this issue in product-tests-basic-environment here but not finding the details around what's causing this. Are there any more logs that get captured as part of workflow run?

agrawalreetika avatar Feb 28 '24 12:02 agrawalreetika

@agrawalreetika why do we need new product tests for this feature? Can't it run in an existing suite?

tdcmeehan avatar Feb 28 '24 14:02 tdcmeehan

@agrawalreetika why do we need new product tests for this feature? Can't it run in an existing suite?

@tdcmeehan I added tests to the existing suite, but looks like product-tests-basic-environment is having same issue here

agrawalreetika avatar Feb 28 '24 19:02 agrawalreetika

@tdcmeehan & @steveburnett all the tests are green, this is ready for final review.

agrawalreetika avatar Mar 06 '24 16:03 agrawalreetika