Syft models should be decoupled from Proto models

[mccorby]

What?

Syft models have a strong dependency on Proto (Protobuffer) data types.

Why?

• Decoupling allow the internal data types in the library and those that map to them to evolve independently.
• Tests will be easier to do.

Breakdown

• Create specific data types that can be mapped to those coming from Proto.
  • For instance, SyftTensort has the following fields:
    • val id: IdOuterClass.Id
    • val contents: TensorDataOuterClass.TensorData

Additional Context

[vkkhare]

So for SyftProtoTensor and SyftTensor, how do you propose to have the fields? I assume instead of setting fields directly you would have a repository that has functions to extract relevant attributes from prototensor? That should be able to decouple the proto and library.

Though how much of an advantage is that decoupling other than testing? Any change in syft proto would need to be propagated through several layers for each of the objects. Considering syft proto to not be backward compatible and rapidly changing I don't know if maintaining these will become tedious?

[mccorby]

Ideally, if we consider Proto as a data source model, we would have a mapper (similar to the current fun SyftProtoTensor.toSyftTensor(). This simple addition would separate both worlds and all changes would happen only in the mapper. This work would require, as you say, modifying the objects up from SyftTensor and others too probably but it would be a one-time change to adapt them to those internal KotlinSyft definitions.

There is the discussion of how to represent those Proto fields/objects.

I do not intend this to be done right now but I did want to keep track of it :)

[mccorby]

"Any change in syft proto would need to be propagated through several layers for each of the objects": This is exactly what this issue tries to solve.

[vkkhare]

Ahh okay so it's removing the companion constructors to extension functions?

[mccorby]

It'll require a bit more of work but those extension functions from proto to KotlinSyft would be part of the solution. Plus, as said, we would need to create our own Id, ContentData, etc...