Report for `Unconstrained` scores

[jnhwkim]

Many recent papers start to report the graph Unconstrained scores.

Two evaluation protocols are used in the literature differing in whether they enforce graph constraints over model predictions. The first graph-constrained protocol requires that the top-K triplets assign one consistent class per entity and relation. The second unconstrained protocol does not enforce any such constraints. (Herzig & Raboh et al., 2018)

My interpretation is Unconstrained scores are relaxed to get a point to match one of the annotations per triplet when calculating Recall@K if there is any in the dataset.

[jwyang]

Good point! in relation detection community, they usually report the unconstrained one. I will try to add this.

From my understanding, in unconstrained setting (relation detection), given N object proposals, we take all Nx(N-1) triplets, and then get [Nx(N-1)] x C scores, for all C predicate categories, then we sort them in descending order, and compute the recall on the ground-truth. However, in the constrained setting (scene graph generation), we take max over [Nx(N-1)] x C for each triplet to get N x (N-1) scores so that each triplet has a single predicate category, and then we sort them and compute the recalls.

[jnhwkim]

1. Is there any reference for this detail? -- I am not familiar with this, so I want to check up. :-)
2. I noticed the recent commits for evaluations and hard-wired print for IMP and motifs. Are you updating the code?

[jnhwkim]

Self-answered to 1. Please see NerualMotif Supplemental. Setup.

Omitting graph constraints, namely, allowing a head- tail pair to have multiple edge labels in system output. We hypothesize that omitting graph constraints should always lead to higher numbers, since the model is then allowed multiple guesses for challenging objects and relations.

Self-answered to 2. Please refer to https://github.com/jwyang/graph-rcnn.pytorch/issues/56.