venice
venice copied to clipboard
[vpj][dvc][cc][test] Create PubSub adapter factories based on VeniceProperties
Problem Statement
Generics are not used correctly in all places in PubSubClientsFactory. This increase chance of class cast exception during runtime.
In TopicMetadataFetcher, PubSubConsumerAdapters are stored in a variable named closeable which is confusing.
In KafkaInputFormat, Kafka admin and consumer adapter factories are created directly without using venice properties. This obstructs using a different type of adapter transparently with the help of properties.
Solution
- added generic statements in PubSubClientsFactory; to achieve this I had to create method
public static PubSubClientsFactory<? extends PubSubProducerAdapter, ? extends PubSubConsumerAdapter, ? extends PubSubAdminAdapter> fromVeniceProperties( VeniceProperties properties) - renamed
closeablestopubSubConsumerAdaptersin TopicMetadataFetcher - used PubSubClientsFactory.fromVeniceProperties to create adaptor factories in KafkaInputFormat
Code changes
- [ ] Added new code behind a config. If so list the config names and their default values in the PR description.
- [ ] Introduced new log lines.
- [ ] Confirmed if logs need to be rate limited to avoid excessive logging.
Concurrency-Specific Checks
Both reviewer and PR author to verify
- [ ] Code has no race conditions or thread safety issues.
- [ ] Proper synchronization mechanisms (e.g.,
synchronized,RWLock) are used where needed. - [ ] No blocking calls inside critical sections that could lead to deadlocks or performance degradation.
- [ ] Verified thread-safe collections are used (e.g.,
ConcurrentHashMap,CopyOnWriteArrayList). - [ ] Validated proper exception handling in multi-threaded code to avoid silent thread termination.
How was this PR tested?
- [ ] New unit tests added.
- [ ] New integration tests added.
- [ ] Modified or extended existing tests.
- [ ] Verified backward compatibility (if applicable).
Does this PR introduce any user-facing or breaking changes?
- [x] No. You can skip the rest of this section.
- [ ] Yes. Clearly explain the behavior change and its impact.
Even for draft PRs, having a clear and descriptive title is helpful.