[Good First Issue][TF FE]: Support complex tensors for Pack operation

[rkazants]

Context

OpenVINO component responsible for support of TensorFlow models is called as TensorFlow Frontend (TF FE). TF FE converts a model represented in TensorFlow opset to a model in OpenVINO opset. Some audio models use tensors of complex type. Complex type tensor is a tensor that has elements of complex type. For example, 1D tensor with three elements x = [1+2j, 2, -2j].

For supporting Pack operation on complex type tensor, you need to extend the corresponding loader for Pack.

What needs to be done?

The existing loader for Pack needs to be extended by propagating ComplexTypeMark from input to output and to represent output complex type tensor as a floating-point type tensor with auxiliary dimension that concatenates real and imaginary parts of complex tensor. To validate the extension, the corresponding layer test needs to be updated with complex tensor cases.

Here is an example of how to extend Reshape loader to support complex type tensors:

OutputVector translate_reshape_op(const NodeContext& node) {
    default_op_checks(node, 2, {"Reshape"}, true);
    auto tensor = node.get_input(0);
    auto complex_type_mark = as_type_ptr<ComplexTypeMark>(tensor.get_node_shared_ptr());
    auto shape = node.get_input(1);
    if (complex_type_mark) {
        element::Type complex_part_type = complex_type_mark->get_complex_part_type();
        tensor = complex_type_mark->input_value(0);

        OutputVector concat_inputs;
        concat_inputs.push_back(shape);
        concat_inputs.push_back(make_shared<v0::Constant>(shape.get_element_type(), Shape{1}, 2));

        auto concat = make_shared<v0::Concat>(concat_inputs, 0);
        auto reshape = make_shared<v1::Reshape>(tensor, concat, false);
        set_node_name(node.get_name(), reshape);
        auto complex_reshape = make_shared<ComplexTypeMark>(reshape, complex_part_type);
        return {complex_reshape->output(0)};
    }

    auto reshape = make_shared<v1::Reshape>(tensor, shape, false);
    set_node_name(node.get_name(), reshape);
    return {reshape};
}

Since OpenVINO does not have native support of complex tensors, we handle complex type in intermediate layers by representing them as a floating-point type with additional dimension (specially created) to store real and imaginary parts of the original complex tensor so slicing by the last dimension will give either real or imaginary parts: x[...,0] - real and x[...,1] - imaginary parts.

On the first step, we update default_op_checks with true flag to indicate that loader for Reshape operation now handles complex tensors:

default_op_checks(node, 2, {"Reshape"}, true);

Secondly, we check if complex type mark exists by anticipated inputs. This mark indicates that input tensor of complex type:

auto complex_type_mark = as_type_ptr<ComplexTypeMark>(tensor.get_node_shared_ptr());

Thirdly, we retrieve a floating-point tensor (with additional dimension to store real and imaginary parts) simulating complex tensor:

tensor = complex_type_mark->input_value(0);

After that, we implement conversion for Reshape for this particular case. Since a floating-point tensor simulating complex tensor has additional dimension equal to 2, we update input target shape by appending 2 value and perform reshape on a floating-point tensor simulating complex tensor.

Finally, since Reshape should produce complex tensor by output we insert a new mark ComplexTypeMark into the output.

To validate support of complex tensors for Reshape, the new layer test TestComplexReshape was added.

Example how to run the layer test:

export TEST_DEVICE=CPU
cd openvino/tests/layer_tests/tensorflow_tests
pytest test_tf_Reshape.py

Example Pull Requests

• Extensions for Shape, Mul, Reshape in https://github.com/openvinotoolkit/openvino/pull/21477
• Extension for Roll operation in https://github.com/openvinotoolkit/openvino/pull/20860

Resources

• What is OpenVINO?
• How to Build OpenVINO
• Developer documentation for TensorFlow Frontend
• Intel DevHub Discord channel - engage in discussions, ask questions and talk to OpenVINO developers

Contact points

• @openvinotoolkit/openvino-tf-frontend-maintainers
• rkazants in Discord

Ticket

No response

[yongyizang]

.take

[github-actions[bot]]

Thank you for looking into this issue! Please let us know if you have any questions or require any help.

[ysrastogi]

.take

[github-actions[bot]]

Thanks for being interested in this issue. It looks like this ticket is already assigned to a contributor. Please communicate with the assigned contributor to confirm the status of the issue.

[ysrastogi]

Hii @yongyizang can we collaborate? I have created the idea and very close to solve the issue.

[yongyizang]

Sure! Sorry, just saw your message.

[yongyizang]

@ysrastogi How do you want to proceed? If you have already created a solution, feel free to proceed without me. I'm open to discussing the issue together if you are looking for collaboration and discussion.

[rkazants]

Hi @yongyizang, any update on this task?

Best regards, Roman

[rkazants]

no update for two weeks. I release this task.

Best regards, Roman

[Pranshu-S]

.take

[github-actions[bot]]

Thank you for looking into this issue! Please let us know if you have any questions or require any help.

[Jessielovecodings]

.take

#WLB

[github-actions[bot]]

Thanks for being interested in this issue. It looks like this ticket is already assigned to a contributor. Please communicate with the assigned contributor to confirm the status of the issue.

[hibahassan1]

.take

[github-actions[bot]]

Thank you for looking into this issue! Please let us know if you have any questions or require any help.

[hibahassan1]

Hello, I have some trouble running the tests. I am working in VS code and am experiencing a lot of ImportErrors. I created a virtual environment and installed all the dependencies, but I am still experiencing this.

[rkazants]

Hello, I have some trouble running the tests. I am working in VS code and am experiencing a lot of ImportErrors. I created a virtual environment and installed all the dependencies, but I am still experiencing this.

Hi,

Please send the error log.

Best regards, Roman

[hibahassan1]

The issue seems to have been fixed by creating a new virtual environment. Thank you