[RFC] 004 operators

[lgarithm]

draft

Draft definitions

A symbolic operator is a generic operator that has a fixed arity. A specialized operator is a tensor function which has a fixed signature. A operator function is a function that takes some configuration parameters and returns a specialized operator.

types of operators

• initializers: are 0-arity operators.
• unary operators: are 1-arity operators.
• binary operators: are 2-arity operators.
• loss functions: binary operators that has generic signature < 0, r, r >.

Example of operators

add

add is a 2-arity operator. add has signature < r, r, r >. add : T_p, T_p \to T_p where p is a shape

mul

mul is a 2-arity operator. mul has signature < r + s, s + t, r + t >. mul : T_(p, q), T_(q, r) \to T_(p, r) where p, q, r can be any shapes. When rank p = rank q = rank r = 1, mul reduces to the matrix times matrix operator, in general cases, mul is known as tensor contraction.

conv

conv is a 2-ariry operator, and can take several shapes as configuration parameters, including padding, stride, sample rate, etc.

pool

pool is a 1-arity operator. pool can take kernal/patch size, stride as configuration parameters, and can take a trait/policy function, which is default to max, and can be mean.

[lgarithm]

draft

The pooling operator

The generic pooling operator with filter shape [r, s] and stride shape [stride_r, stride_s] (TODO: add padding shape) has the following signature:

<3, 3> or <4, 4>

pool : [n, h, w, c] -> [n, h', w', c]

where n is the batch size, c is the channel size h' = (h - r) / stride_r + 1 w' = (w - s) / stride_s + 1

[lgarithm]

draft

Pointwise operators

Pointwise operators are generic unary operators that has generic signature <r, r>

common pointwise operators are

• relu
• leaky relu

[lgarithm]

Other topics:

• #35: Affine operator
• #40: Support split and merge operators
• #60: Operator fusion
• #62: Variadic operator