venice
venice copied to clipboard
[VPJ] Add optional top level timestamp record to VPJ
This allows users to set a top level timestamp on a record in the input
Problem Statement
We support record level timestamps in nearline and in incremental push, but we don't have it in the form of batch.
Solution
We tweak the schema slightly for record input. Previously this took a key and a value. This adds a fourth field called 'timestamp' which a user can specify per record optionally. If a user doesn't specify a timestamp, then one is not transmitted to Venice.
Code changes
- [ ] Added new code behind a config. If so list the config names and their default values in the PR description.
- [ ] Introduced new log lines.
- [ ] Confirmed if logs need to be rate limited to avoid excessive logging.
Concurrency-Specific Checks
Both reviewer and PR author to verify
- [ ] Code has no race conditions or thread safety issues.
- [ ] Proper synchronization mechanisms (e.g.,
synchronized,RWLock) are used where needed. - [ ] No blocking calls inside critical sections that could lead to deadlocks or performance degradation.
- [ ] Verified thread-safe collections are used (e.g.,
ConcurrentHashMap,CopyOnWriteArrayList). - [ ] Validated proper exception handling in multi-threaded code to avoid silent thread termination.
How was this PR tested?
- [ ] New unit tests added.
- [ ] New integration tests added.
- [ ] Modified or extended existing tests.
- [ ] Verified backward compatibility (if applicable).
Does this PR introduce any user-facing or breaking changes?
- [ ] No. You can skip the rest of this section.
- [ ] Yes. Clearly explain the behavior change and its impact.