[SPARK-38909][CORE][YARN] Encapsulate `LevelDB` used to store remote/external shuffle state as `DB`

[LuciferYang]

What changes were proposed in this pull request?

ExternalShuffleBlockResolver, YarnShuffleService and RemoteBlockPushResolver use LevelDB directly, this is not conducive to extending the use of RocksDB in this scenario. This pr is encapsulated for expansibility. It will be the pre-work of SPARK-38888, the main change as follows:

• Introduce DB interface and implementation of corresponding LevelDB now, RocksDB implementation will be involved in the future
• Introduce DBBackend enum and new config SHUFFLE_SERVICE_DB_BACKEND to specify a disk-based store used in shuffle service local db, only LEVELDB is supported now, RocksDB implementation will be involved in the future
• Introduce DBIterator to traverse the items in the DB, only LevelDBIterator is supported now, RocksDBIterator implementation will be involved in the future
• Introduce DBProvider to initialization DB, it is only used to generate LevelDB now, RocksDB implementation will be involved in the future
• Change StoreVersion to top-level class, it will be reused by RocksDB implementation
• Replace the directly use of LevelDB in ExternalShuffleBlockResolver, YarnShuffleService and RemoteBlockPushResolver with DB
• fix corresponding UTs and test tools

Why are the changes needed?

This is pre work of SPARK-38888

Does this PR introduce any user-facing change?

• Add new config SHUFFLE_SERVICE_DB_BACKEND(spark.shuffle.service.db.backend), the new config use to use to specify a disk-based store used in shuffle service local db. Only LEVELDB is supported after this pr.

How was this patch tested?

Pass GA

[dongjoon-hyun]

Although I couldn't take a look at this, you can ask YARN committers' help if you want to make this proceed, @LuciferYang .

[LuciferYang]

thanks @dongjoon-hyun

[LuciferYang]

I found RemoteBlockPushResolver begin to use LevelDB, more refactor work is required before reopen this pr

[dongjoon-hyun]

Got it. That's bad.

[mridulm]

This is a recent change, to add missing functionality for preserving merge data across restarts. Please tag me when you have an updated PR @LuciferYang, will help review it.

[LuciferYang]

Merge with master and re-trigger GA

[LuciferYang]

This is a recent change, to add missing functionality for preserving merge data across restarts. Please tag me when you have an updated PR @LuciferYang, will help review it.

@dongjoon-hyun @mridulm Conflict is resolved and GA passed. But I think we should make sure that SPARK-38888 is worth to do before reviewing, because this pr is the pre-work of SPARK-38888.

[mridulm]

@dongjoon-hyun @mridulm Conflict is resolved and GA passed. But I think we should make sure that SPARK-38888 is worth to do before reviewing, because this pr is the pre-work of SPARK-38888.

I agree with @dongjoon-hyun's assessment in the jira, and I dont have strong opinion. If rest of the codebase is moving towards an abstraction for db, I would prefer we did the same for yarn shuffle as well though from a consistency point of view.

[dongjoon-hyun]

Actually, instead of providing wrapper layer, the final goal is the removal of LevelDB in favor of RocksDB.

[mridulm]

@dongjoon-hyun

Actually, instead of providing wrapper layer, the final goal is the removal of LevelDB in favor of RocksDB.

I am sure in future, we might have another SPIP to remove RocksDB in favor of something else :-) If we can abstract it away, the future evolution might be easier !

[mridulm]

nit: Btw, from a completeness point of view, DB.get makes sense to include :-)

[LuciferYang]

nit: Btw, from a completeness point of view, DB.get makes sense to include :-)

OK ~

[LuciferYang]

Also cc @tgravescs do you have time to help review this pr? Thanks ~

[tgravescs]

changes look fine to me

[mridulm]

Merged to master. Thanks for working on this @LuciferYang ! Thanks for reviews @dongjoon-hyun, @zhouyejoe and @tgravescs :-)

[LuciferYang]

Thanks @mridulm @tgravescs @zhouyejoe @dongjoon-hyun !!! I will do the RocksDB related work next week due to some other work needs to be completed this week ~