florencejt
Extending and generalising to other modalities
Discussed in https://github.com/florencejt/fusilli/discussions/20
Originally posted by mdkozlowski January 14, 2024 Hi, For my usecases I'm interested in some additional modalities that aren't currently supported in fusili, such as graph-structured data (as inputs) and text data. Fusion of multiple modalities on a graph input (such as tabular + textual features, per-node) is specifically interesting. On the other hand, my usecase doesn't make use of images or image models.
If it makes sense and depending on interest, I'd be happy to contribute these kinds of models to the project.
At the moment the dataloaders and data classes are quite specific to combinations of tabular & images. Do you see any value in making the the data classes more generic? For example, removing dependencies on image_downsample_size in TrainTestDataModule, and using naming in the project like embedding or dense_representation - agnostic to the embedding modality.
Definitely an issue to work on down the line to make fusilli not just image focused. It will require some refactoring of the data loading portion so I think this will be something for later this year.