Possibility to represent the color as given from a classifier

[dfahlen]

It would be good if it was possible to represent the color as given by a classifier.

An attempt to address this was done in #213 by adding a likelihood attribute to the Color type used in the ColorCluter type. This may have been a somewhat naive solution as it may lead to some confusion regarding the originally intended use of the ColorCluster type. For this reason, this issue was created to discuss the topic in more detail, give some background and discuss possible solutions.

Background

The classifier will generate the likelihood that an object is of a particular color from a list of colors that it has been trained on. Multiple colors may be correct.

For example, suppose the classifier has been trained on the colors [Red, Green, Blue, White, Black]. If the classifier is used on a car that is completely blue it may result in a list of likelihoods that looks like this:

Color	Likelihood
Blue	0.91
Black	0.14
Green	0.12
White	0.07
Red	0.05

If the classifier is used on a car that is blue with a white roof it may result in a list of likelihoods that looks like this:

Color	Likelihood
Blue	0.93
White	0.87
Black	0.12
Green	0.08
Red	0.04

There is no clear way to represent this using the ColorCluster type as it is currently specified. For it to be possible to represent this in ONVIF the spec needs to be extended with:

• A color representation with a likelihood
• A way to represent multiple color candidates with different likelihoods

Possible solutions

Below are two possible solutions to solve the above use case.

Suggestion 1

This suggestion uses the same solution as in #213 but with some clarifications regarding the use of the likelihood attribute.

In Analytics.xml under the existing paragraphs in the Color descriptor section add something like this:

An alternative representation of color can be used by adding a likelihood attribute to the color cluster.

Its likelihood denotes how confident the algorithm is that the color is correct. The range of likelihood is [0 .. 1].

When using the likelihood attribute with the color cluster the weight and covariance should not be used as they are missing a meaningful definition in this case.

When using the likelihood attribute the meaning of multiple color clusters in the same color descriptor changes. Multiple color clusters with likelihood should be interpreted as different color candidates with different likelihoods, not as different regions of colors on the object.

An example metadata containing color information of the detected object using a color cluster with a likelihood attribute is given below.

<tt:Frame UtcTime="2010-09-10T12:24:57.721" Colorspace="RGB">
  <tt:Transformation>
    <tt:Translate x="-1.0" y="-1.0"/>
    <tt:Scale x="0.003125" y="0.00416667"/>
  </tt:Transformation>
  <tt:Object ObjectId="34">
    <tt:Appearance>
      <tt:Shape>
        <tt:BoundingBox left="20.0" top="80.0" right="100.0" bottom="30.0"/>
        <tt:CenterOfGravity x="60.0" y="50.0"/>
      </tt:Shape>
      <tt:Color>
        <tt:ColorCluster>
          <tt:Color X="0" Y="0" Z="255" Likelihood="0.93"/>
        </tt:ColorCluster>
        <tt:ColorCluster>
          <tt:Color X="255" Y="255" Z="255" Likelihood="0.87"/>
        </tt:ColorCluster>
        <tt:ColorCluster>
          <tt:Color X="0" Y="0" Z="0" Likelihood="0.12"/>
        </tt:ColorCluster>
        <tt:ColorCluster>
          <tt:Color X="0" Y="128" Z="0" Likelihood="0.08"/>
        </tt:ColorCluster>
        <tt:ColorCluster>
          <tt:Color X="255" Y="0" Z="0" Likelihood="0.04"/>
        </tt:ColorCluster>
      </tt:Color>
    </tt:Appearance>
  </tt:Object>
</tt:Frame>

In common.xsd add:

<xs:complexType name="Color">
  <xs:attribute name="X" type="xs:float" use="required"/>
  <xs:attribute name="Y" type="xs:float" use="required"/>
  <xs:attribute name="Z" type="xs:float" use="required"/>
  <xs:attribute name="Colorspace" type="xs:anyURI">
    <xs:annotation>
      <xs:documentation>
        Acceptable values:
        <ul>
          <li>http://www.onvif.org/ver10/colorspace/YCbCr - YCbCr
            <ul><li>X attribute = Y value</li>
              <li>Y attribute = Cb value</li>
              <li>Z attribute = Cr value</li></ul>
          </li>
          <li>http://www.onvif.org/ver10/colorspace/RGB - RGB
            <ul><li>X attribute = R value</li>
              <li>Y attribute = G value</li>
              <li>Z attribute = B value</li></ul>
          </li>
        </ul>
        If the Colorspace attribute is absent and not defined on higher level, YCbCr is implied.
        Deprecated values:
        <ul>
          <li>http://www.onvif.org/ver10/colorspace/CIELUV - CIE LUV</li>
          <li>http://www.onvif.org/ver10/colorspace/CIELAB - CIE 1976 (L*a*b*)</li>
          <li>http://www.onvif.org/ver10/colorspace/HSV - HSV</li>
        </ul>
      </xs:documentation>
    </xs:annotation>
  </xs:attribute>
  <xs:attribute name="Likelihood" type="xs:float">
    <xs:annotation>
      <xs:documentation>Likelihood that the color is correct.</xs:documentation>
    </xs:annotation>
  </xs:attribute>
</xs:complexType>

Some pros and cons of this solution are:

• Pro and Con: Perceived compatibility with implementations of the current color spec.
• Con: Color cluster is used as a color candidate instead of a color region when the likelihood attribute is used, which may be confusing.
• Con: May break implementations expecting color clusters to be used as color regions, not as color candidates.
• Con: It would be confusing if colors with and without likelihood were used in the same color descriptor. E.g. mixing the color candidate and a color region concepts.

Suggestion 2

This is a solution that is more in line with how data with a likelihood is presented in the rest of the standard.

In Analytics.xml under the existing paragraphs in the Color descriptor section add something like this:

An alternative representation of color is color candidates. It is an orthogonal concept to color clusters and may be used together with color clusters or as a stand-alone representation of the color.

A color can be described by a set of color candidates. The color candidates are a list of color names together with a likelihood that describes how confident the algorithm is that the color is correct. The range of likelihood is [0 .. 1].

Note that it is not specified how color candidates are created but a common case maybe by using a classifier algorithm.

Besides the set of predefined color candidates (based on the css colors see: https://www.w3schools.com/cssref/css_colors.asp) free type definitions can be added using the ColorType member defined as StringLikelihood.

An example metadata containing color information of the detected object using color candidates is given below.

<tt:Frame UtcTime="2010-09-10T12:24:57.721">
  <tt:Transformation>
    <tt:Translate x="-1.0" y="-1.0"/>
    <tt:Scale x="0.003125" y="0.00416667"/>
  </tt:Transformation>
  <tt:Object ObjectId="34">
    <tt:Appearance>
      <tt:Shape>
        <tt:BoundingBox left="20.0" top="80.0" right="100.0" bottom="30.0"/>
        <tt:CenterOfGravity x="60.0" y="50.0"/>
      </tt:Shape>
      <tt:Color>
        <tt:ColorType Likelihood="0.93">Blue</ColorType>
        <tt:ColorType Likelihood="0.87">White</ColorType>
        <tt:ColorType Likelihood="0.12">Black</ColorType>
        <tt:ColorType Likelihood="0.08">Green</ColorType>
        <tt:ColorType Likelihood="0.04">Red</ColorType>
      </tt:Color>
    </tt:Appearance>
  </tt:Object>
</tt:Frame>

In common.xsd add:

<xs:simpleType name="ColorType">

    <xs:restriction base="xs:string">
        <xs:enumeration value="Red"/>
        <xs:enumeration value="Green"/>
        <xs:enumeration value="Blue"/>
        .... all 140 color names in the css color standard
    </xs:restriction>

</xs:simpleType>

<xs:complexType name="ColorDescriptor">

		<xs:sequence>
			<xs:element name="ColorCluster" minOccurs="0" maxOccurs="unbounded">
				<xs:complexType>
					<xs:sequence>
						<xs:element name="Color" type="tt:Color"/>
						<xs:element name="Weight" type="xs:float" minOccurs="0"/>
						<xs:element name="Covariance" type="tt:ColorCovariance" minOccurs="0"/>
						<xs:any namespace="##any" processContents="lax" minOccurs="0" maxOccurs="unbounded"/>	<!-- reserved for ONVIF -->
					</xs:sequence>
				</xs:complexType>
			</xs:element>
      <xs:element name="ColorType" type="tt: StringLikelihood" minOccurs="0" maxOccurs="1">  <!-- tt:ColorType lists the acceptable values" -->
			<xs:element name="Extension" type="xs:anyType" minOccurs="0"/>
			<xs:any namespace="##any" processContents="lax" minOccurs="0" maxOccurs="unbounded"/>	<!-- reserved for ONVIF -->
		</xs:sequence>
		<xs:anyAttribute processContents="lax"/>
	</xs:complexType>

Some pros and cons of this solution are:

• Pro: Intuitive color representation when using classifiers.
• Pro: Clear separation of the concepts of color clusters and color candidates.
• Pro: Supports both concepts of color regions/clusters and color candidates
• Pro: Similar to a class type, vehicle type, etc.
• Con: Probably a good idea to have both candidates and a cluster for compatibility with existing implementations.
• Con: There can be two parallel color representations that an implementation must choose from.

Conclusion

It is not trivial to solve this use case.

Is this a valid use case? Should it be possible to present color candidates with a likelihood? Which of the solutions is best? Is there any other solution that may be better?

[svefredrik]

Feedback from Enhancements WG: Suggestion 1 is the only viable, backward compatible solution. Please update PR with this. Make it two different examples, one with Weight and one with Likelihood.

[kieran242]

I agree with Fredrik and have reviewed and commented on PR#213 already.

[HansBusch]

For progress see PR #213