[FLINK-36540][Runtime] Add Support for Hadoop Caller Context when using Flink to operate hdfs.

[liangyu-1]

What is the purpose of the change

As described in FLINK-36540. When we use Flink to delete or write or modify files on Hadoop filesystem, callerContext is a helpful feature if we want to trace who did the operation or count how many files an application can create on hadoop filesystem. UGI is not good enough to trace these operations because if we have a tenant who has a lot of jobs writing into HDFS, we cannot find out which job caused the breakdown of HDFS.

I created a new interface and class in flink-core module, so that it will not cause the leak in ThreadLocal value, and it won't influence the situation if we do not use hdfs.

What's more, with this new feature and history json files in history server, we can calculate how many read operations and write operations a Flink application did to hdfs, and find out if there is a pressure or bottleneck to operate on hdfs files.

Brief change log

• Add a new interface ContextWrapperFileSystem
• Add a new class FileSystemContext
• Add a new class HadoopFileSystemWithContext
• Add initialization operation at the place where we initialize FileSystemSafetyNet

Verifying this change

Please make sure both new and modified tests in this PR follow the conventions for tests defined in our code quality guide.

This change added tests and can be verified as follows:

(example:)

• Tested on our YARN CLUSTER

I rebuild this project, and test the new jar file in my cluster, it prints out the correct caller context as expected

Does this pull request potentially affect one of the following parts:

• Dependencies (does it add or upgrade a dependency): (yes / no)
• The public API, i.e., is any changed class annotated with @Public(Evolving): (yes / no)
• The serializers: (yes / no / don't know)
• The runtime per-record code paths (performance sensitive): (yes / no / don't know)
• Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know)
• The S3 file system connector: (yes / no / don't know)

Documentation

• Does this pull request introduce a new feature? (yes / no)
• If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

[flinkbot]

CI report:

• 89f262864a8ee1084f5b896c1763539a2156ec3c Azure: SUCCESS

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot run azure re-run the last Azure build

[liangyu-1]

@dmvk @xintongsong @ferenc-csaky Hi, would you please help me check this PR? I implement this feature in a new way which is created in https://github.com/dmvk/flink/commit/bfe9f6082518c19808244fe27aedaba79c5dbb74

[liangyu-1]

@flinkbot run azure

[ferenc-csaky]

Does the CI failure related to this change? If not, let's rebase it to the latest master.

[liangyu-1]

thanks for your reply @ferenc-csaky I have re-run the failed UT, and it passed, so I think the failure is not related to this change.

[liangyu-1]

@twalthr Hi, would you please help me check this pr?