venice icon indicating copy to clipboard operation
venice copied to clipboard

[vpj][dvc][cc][test] Create PubSub adapter factories based on VeniceProperties

Open arjun4084346 opened this issue 8 months ago • 1 comments

Problem Statement

Generics are not used correctly in all places in PubSubClientsFactory. This increase chance of class cast exception during runtime. In TopicMetadataFetcher, PubSubConsumerAdapters are stored in a variable named closeable which is confusing. In KafkaInputFormat, Kafka admin and consumer adapter factories are created directly without using venice properties. This obstructs using a different type of adapter transparently with the help of properties.

Solution

  • added generic statements in PubSubClientsFactory; to achieve this I had to create method public static PubSubClientsFactory<? extends PubSubProducerAdapter, ? extends PubSubConsumerAdapter, ? extends PubSubAdminAdapter> fromVeniceProperties( VeniceProperties properties)
  • renamed closeables to pubSubConsumerAdapters in TopicMetadataFetcher
  • used PubSubClientsFactory.fromVeniceProperties to create adaptor factories in KafkaInputFormat

Code changes

  • [ ] Added new code behind a config. If so list the config names and their default values in the PR description.
  • [ ] Introduced new log lines.
    • [ ] Confirmed if logs need to be rate limited to avoid excessive logging.

Concurrency-Specific Checks

Both reviewer and PR author to verify

  • [ ] Code has no race conditions or thread safety issues.
  • [ ] Proper synchronization mechanisms (e.g., synchronized, RWLock) are used where needed.
  • [ ] No blocking calls inside critical sections that could lead to deadlocks or performance degradation.
  • [ ] Verified thread-safe collections are used (e.g., ConcurrentHashMap, CopyOnWriteArrayList).
  • [ ] Validated proper exception handling in multi-threaded code to avoid silent thread termination.

How was this PR tested?

  • [ ] New unit tests added.
  • [ ] New integration tests added.
  • [ ] Modified or extended existing tests.
  • [ ] Verified backward compatibility (if applicable).

Does this PR introduce any user-facing or breaking changes?

  • [x] No. You can skip the rest of this section.
  • [ ] Yes. Clearly explain the behavior change and its impact.

arjun4084346 avatar Apr 17 '25 01:04 arjun4084346

Even for draft PRs, having a clear and descriptive title is helpful.

sushantmane avatar Apr 23 '25 19:04 sushantmane