Implementing Neural Networks

[CSSFrancis]

Describe the functionality you would like to see.

There is lots of good work being done with neural networks for processing data and it would be good to think about how we would add some of that functionality to hyperspy.

Describe the context

This is a non-complete list but there are lots of people doing some really good work with neural networks in the microscopy community.

https://github.com/elisathr/SPED-phase-mapping https://github.com/flysteven/ANN-4DDD https://github.com/AI-ML-4DSTEM/crystal4D https://github.com/wdwzyyg/ElectronCounting https://github.com/Ivanlh20/r_em

Additional information

Unfortunately there appears to be a fairly big disconnect between training some model and then applying the model. Most of the models could be easily applied using the map function. It might be sufficient to just include an example of how someone might use the map function to apply a NN.

The important part is to have it be easy to apply a neural network so some end user can just try it without too much overhead (My understanding is that this is not really that easy).

@magnunor Do you have any other thoughts? Your group has been playing around with this a little so you might have some good ideas.

[francisco-dlp]

@CSSFrancis, nice proposal.

I fully agree with your stated goal "to have it be easy to apply a neural network so some end user can just try it without too much overhead". The current learning features of HyperSpy , while popular, are too inflexible and, since the dawn of NN (a while ago), out of touch with modern machine learning. Therefore, I think that there is scope of a hyperspy-learn package that offers a more flexible interface to the current learning features and adds NN features that are mature enough for general consumption. Eventually, we could remove the learning features from HyperSpy 3.0.

[magnunor]

We've mostly been training models for analyzing magnetic STEM-DPC data, and testing this on "local" datasets. But it would be really nice to make it easier for other people to utilize the models we've trained.

@sivborg and @grigoryn94 should have some input

[CSSFrancis]

I fully agree with your stated goal "to have it be easy to apply a neural network so some end user can just try it without too much overhead". The current learning features of HyperSpy , while popular, are too inflexible and, since the dawn of NN (a while ago), out of touch with modern machine learning. Therefore, I think that there is scope of a hyperspy-learn package that offers a more flexible interface to the current learning features and adds NN features that are mature enough for general consumption. Eventually, we could remove the learning features from HyperSpy 3.0.

I think the point about being out of touch with modern machine learning is a good one to consider. I don't know if hyperspy should really be in the business of writing code for machine learning alone. With the limited number of developers it seems like things start to fall by the wayside and writing good ml code is a big ask. That being said dask-ml still doesn't have a nnmf implementation so maybe depending on other packages for development isn't the best idea and there is a real need in the python community for a central ml package focused on high dimensional data. The scope of that problem seems large and doing it right seems difficult without more support from a package like 'daskorscipy` etc.

I think in an ideal world some machine learning package would:

Support Basic Matrix Factorization:

• PCA https://ml.dask.org/modules/generated/dask_ml.decomposition.PCA.html
• SVD https://ml.dask.org/modules/generated/dask_ml.decomposition.TruncatedSVD.html
• NNMF

Support Tensor Based Matrix factorization

• Tensor Based SVD https://github.com/paul-voyles/tensor_svd
• Tensor Based NNMF https://icml.cc/Conferences/2005/proceedings/papers/100_NonNegative_ShashuaHazan.pdf https://github.com/drumichiro/nmf-and-ntf

Support Neural Networks:

• Support training neural networks based on labeled data in hyperspy (potentially)
• Support applying some trained model to some data
• Visualizing and analyzing results.

I think hyperspy actually fits really cleanly in the "visualization" front. If we want to carve out a niche there aren't many multi-dimensional image viewer that run smoothly embedded in the browser. The ability to run remotely, run potentially in jupyter-lite, plot statically and dynamically without dealing with qt are all strong suits for hyperspy and things I haven't really seen replicated elsewhere. Machine learning could potentially be another thing that hyperspy does well but we would probably need a couple of students actively working on making that happen.

For now maybe we just add a section about applying a neural network using map? And maybe something about visualizing NN results?

[sivborg]

Regarding the sharing of pretrained models, which is part of what we're doing, I can come with some input. To share a model, you need the model weights and the structure of the neural network. The weights depend on the network, and can be very large (~240 MB for Resnet 152), though for our more simple networks they are around 10 MB. So whether we want to bundle this in HyperSpy is an important question. Though one possibility is to make them available on-demand through a URL.

The model structure is coded with PyTorch, so the user would need to download this, as far as I've understood. It is a huge library, so perhaps the best way to incorporate this is to bundle the model files, but let the user themselves download PyTorch if they want to use them.

The training of neural networks would require a lot of work to implement and maintain, and would probably quickly be surpassed in advances by more specialized ML libraries. But the use of pretrained networks and visualizing the results are more tractable.