[RFC] DIVE JSON Schema v2

[subdavis]

This is an RFC for version 2 of the DIVE json format.

Option 1: model as tables

import { TrackData, StringKeyObject, ConfidencePair } from 'src/tracks';

interface GroupData {
  groupId: number;
  attributes: StringKeyObject;
  confidencePairs: Array<ConfidencePair>;
  members: Record<TrackId, {
    ranges: number[][]>;
  }>;
  begin?: number;
  end?: number;
  meta?: StringKeyObject;
}

/** For comparison */
type DiveSchemaV1 = Record<String, TrackData>;

interface DiveSchemaV2 {
  version: number;
  tracks: Record<string, TrackData>;
  groups: Record<string, GroupData>;
}

GroupData

A grouping can be used to describe a MEVA activity, but is not tightly coupled.

• By default, it has no begin and end time: groupings implicitly exist only as long as their member tracks exist. Begin and end can, however, be provided if necessary.
• Groups can also have confidence values that work conceptually the same as track confidencePairs.

Group membership modeling

Here's an example membership description for a group with 3 track members.

const groupdata: GroupData = {
  members: {
    "1": {},
    "2": { ranges: [2, 5] },
    "3": { ranges: [3, 4, 6, 9] },
  },
}

• Track 1 is implicitly included for the whole group duration.
• Track 2 is present from frame 2 to frame 5
• Track 3 is present from 3 to 4, leaves, and comes back from 6 to 9.

Members values are objects (instead of simply arrays) to allow for forward compatibility with new information that may need to exist about the membership, such as confidence or provenance (i.e. "track 3 is necessarily a member because it's an object being carried by track 2, which is a member").

Normalization

This schema seeks to normalize information between tracks and groups. The best example of this is group duration. While a user is annotating, they may add new frames to the end of the track, increasing its duration, and thereby possibly increasing the duration of any groups it is a member of. It should not be necessary in code to continually update a group's overall duration. Instead, the duration will be a dynamically computed property.

Implementation

It appears to me that useTrackStore is already generic enough that it would accept either tracks or groups as written, so we might just be able to rename that module and have:

const trackStore = useEntityStore();
const activityStore = useEntityStore();

Option 2: Just unique identifiers

We wouldn't need to change the annotation file format. We could just add { type: group } into the group schema and let our Record<string, TrackData> become Record<string, TrackData | GroupData>

Option 3: Supertracks

Same as above, but we merge the schema for tracks and groups in a backward-compatible way. Allow tracks to have a new "members" property, such that tracks can be group-like (where they have no features of their own, but have member tracks) or track-like (no members, just features) or supertrack-like where they have both. An example use case is compound disconnected annotations: 5 rectangles that describe a posed person, for example. These supertracks could be arbitrarily nested without impacting performance.

From the data side, this would be a tiny change: we only have to add a new optional members field to the track schema.

[BryonLewis]

I need a bit more time to digest and thing about this but wanted to write this down before I forgot.

I feel a bit hesitant about Option 2 and Option 3 mostly because of the mixing of things that seem to have different use cases even if their structures are similar. Granted data schema doesn't need to reflect the in-memory representation but one of the benefits of JSON is that it can be close.