flink-cdc
flink-cdc copied to clipboard
[FLINK-36406] Close MetadataApplier when the job stops
This pull request implements closing MetadataApplier
when the job stops. This allows MetadataApplier
implementations to release the resources that they may use to apply schema changes (e.g. a JDBC connection).
Choice of the interface
-
SchemaRegistryRequestHandler
(which will closeMetadataApplier
) implementsCloseable
, so it would make sense to declareMetadataApplier
asCloseable
as well. -
PaimonMetadataApplier
internally instantiates a PaimonCatalog
, which it ideally should close, and which isAutocloseable
.
To me, it makes more sense to declare MetadataApplier
as AutoCloseable
. The reason is that Closeable
is an IO-specific interface, which is declared in java.io
and throws IOException
in close()
. AutoCloseable
is more generic and seems to be more suitable for MetadataApplier
, whose implementations doesn't necessarily perform IO.
Testing
I didn't find existing tests focusing on SchemaRegistryRequestHandler
or SchemaRegistry
, so I dind't implement a unit test. Testing of the changes in SchemaRegistryRequestHandler
can be done by implementing some logging in ValuesMetadataApplier#close
and running any test that uses it (e.g. FlinkPipelineComposerITCase
)
Additional scope
Just as a demonstration of where this new functionality can be applied, I implemented close()
in PaimonMetadataApplier
. I didn't test it and am not sure that this is the right thing to do. I can revert this part, if necessary.
Further considerations
The problem of closing resources seems also relevant for the MetadataAccessor
interface. Specifically, MySqlMetadataAccessor
disconnects from the database after each call but some other implementations may require maintaining a persistent connection.