Release 1.7.2 and 2.0.0

[ericpre]

v1.7.2

There are several bug fixes in the RELEASE_next_patch branch and it would be good to make a quick patch release

• [x] https://github.com/hyperspy/hyperspy/pull/3008
• [x] https://github.com/hyperspy/hyperspy/pull/3015
• [x] https://github.com/hyperspy/hyperspy/pull/3016

v2.0.0

As the IO have been split - see https://github.com/hyperspy/hyperspy/pull/2972 and https://github.com/hyperspy/rosettasciio - we should get 2.0 released ASAP.

As this is major release, it is a good opportunity to change some API

• [ ] Clear all deprecations
  • [x] #2998
  • [x] #2999
  • [x] #3000
  • [x] #3007
  • [x] #3014
  • [x] #3017
  • [ ] Remove components deprecation
  • [ ] Check for leftovers of the deprecation (especially in the documentation)
• [ ] go thought some API changes worth doing - for example https://github.com/hyperspy/hyperspy/issues/1227 - rename find_peaks1D_ohaver to find_peaks so that it matches its Signal2D counterpart? - rename crop_image to crop_signal2D to be consistent with crop_signal1D? - reorganize metadata: https://github.com/hyperspy/hyperspy/issues/2095
• [ ] Clarify what is public/private, and improve API documentation? - See for example: https://github.com/hyperspy/hyperspy/pull/3014#discussion_r973575789
• [ ] Migrate deprecated Traits on_trait_change() to new observe(): https://docs.enthought.com/traits/traits_user_manual/notification.html#migration-from-on-trait-change
• [ ] #2876?
• [ ] #2997
• [ ] Check with libraries relying on hyperspy that they are happy with the 2.0 release
• [ ] Review release notes, particularly consistency in the API removal, etc.

Anything else, which would be worth considering, please edit or add a comments in this issue!

[hakonanes]

Check with libraries relying on hyperspy that they are happy with the 2.0 release

We haven't tested kikuchipy with HyperSpy 2.0, but I will report back here once we have. Is there anything special we should be aware of?

[CSSFrancis]

@ericpre There are a couple of libraries that would benefit from hyperspy/hyperspy#2876. I'd like for the find_peaks method to just return a vector signal. As written there shouldn't be many conflicts down stream but it is probably best to make that change during the major release.

Pyxem is trying to release a 1.0.0 version as well so my hope is that will closely follow the 2.0 release here.

As far as other things to consider. I think in hyperspy/hyperspy#821 there was also some talk about splitting off the EDS and EELS signals and models. This might be worth a discussion as well. I feel it might be hurting our chances of being used as a general high dimensional image viewer.

[CSSFrancis]

It might also be a good idea to formalize how Hyperspy handles parallelization. At the very least I think that this involves more documentation about how to use dask as well as a discussion on the different schedulers. Currently, every function calling map is parallel by default because dask is parallel by default even if parallel=False. (Some call this a bug, some call this a feature!)

I think this is a good thing as it gives the user the ability to set up dask however works for them (or not set it up and have it just work by default).

The only other place where multiprocessing comes in is with the SamFIRE utils. I've given those a small look, and think that switching those to dask might also be possible. That would simplify the code a fair amount and really streamline all more advanced operations through dask rather than relying on custom built parallelization code.

I'll create an issue for this and link it here. (https://github.com/hyperspy/hyperspy/issues/2997)

[ericpre]

As far as other things to consider. I think in https://github.com/hyperspy/hyperspy/issues/821 there was also some talk about splitting off the EDS and EELS signals and models. This might be worth a discussion as well. I feel it might be hurting our chances of being used as a general high dimensional image viewer.

This is definitely something that will happen but I don't know if this would be possible in a reasonable timeframe for 2.0.

I'll create an issue for this and link it here. (https://github.com/hyperspy/hyperspy/issues/2997)

Thanks @CSSFrancis for this suggestion, indeed, it would be to improve it.

[ericpre]

Check with libraries relying on hyperspy that they are happy with the 2.0 release

We haven't tested kikuchipy with HyperSpy 2.0, but I will report back here once we have. Is there anything special we should be aware of?

@hakonanes, it may be better to check after the deprecation and api changes have been completed.

[francisco-dlp]

Major releases force people to update their codes and scripts, so we should minimize their number. Therefore, I think that it would be good to split the TEM features into a different package for the 2.0 release. It shouldn't be too hard. I think that splitting them into a single package would be good. What do others think? The biggest difficulty is to find a name for such a package.

[CSSFrancis]

Major releases force people to update their codes and scripts, so we should minimize their number. Therefore, I think that it would be good to split the TEM features into a different package for the 2.0 release. It shouldn't be too hard. I think that splitting them into a single package would be good. What do others think? The biggest difficulty is to find a name for such a package.

I'm in support of it. In terms of names I like the direction the Lumispy took in keeping the -spy as that suggests the connection between the package and hyperspy.

So maybe electrospy, elspy or espy.

Spectrospy? Spectspy

[hakonanes]

Realize I might have less to say on the matter since I’m not a core developer of HyperSpy, but keeping the HyperSpy name for the spectroscopy package and calling the core package HyperSpy-core/base is also an option.

[ericpre]

Splitting all domain-specific signals would be great. A few comments:

• I don't think that it would be possible to complete https://github.com/hyperspy/hyperspy/pull/2836 in a reasonable timeframe, so this PR may need to be redone after the split.
• HologramImage is in principle not limited to electron microscopy, so it may be better that it goes to a separate package hyperspy-holo.
• A good word to describe both EDS and EELS without mentioning them explicitly is "analytical electron microscopy".

[jlaehne]

Splitting all domain-specific signals would be great. A few comments: Yes, I also wanted to bring up these additional splits as well. I think it makes totally sense to do them now (even though I am not happy delaying the 2.0 release as I want luminescence signals from rosettascio to be available to lumispy through v2.0)

* `HologramImage` is in principle not limited to electron microscopy, so it may be better that it goes to a separate package `hyperspy-holo`.

Yes, I think holography is something completely different than EELS and EDX.

* A good word to describe both EDS and EELS without mentioning them explicitly is "analytical electron microscopy".

However, for me also CL and EBSD are analytical electron microscopy, but already contained in separate extensions.

So maybe electrospy, elspy or espy. I like ElSpy or ElectronSpy or even eSpy

[ericpre]

I think this is time to decide if we split into 2 or 3 libraries:

3 libraries (vote :rocket: )	2 libraries (vote :tada:)
hyperspy	hyperspy
spectroscopy (EDS and EELS)	spectroscopy (EDS and EELS) and holography
holography

Going for the 3 libraries option has the advantage of having the scope of the package well defined, while having holography in the same library as EELS and EDS sounds like an after thought. There will be some overhead with maintaining one more library, but we are well experienced with having a lot of the testing and packaging automated, so it will not be much. And we never know, as some point, some people may be interested to develop the holography library and this could be advertised as such: "maintenance only and looking for developer".

This also has the advantage of putting spectroscopy techniques into a same library and this would good to be able to add XRF signal to this spectroscopy package - if we go for the 2 libraries option, it may end up being focused on electron based technique and exclude XRF, for example. This has been discussed several times already and there are many similarities between EDS and XRF.

As silver lining, we can use the -spy pattern (see https://github.com/hyperspy/hyperspy/issues/2996#issuecomment-1228504094), such as holospy and spectrospy are available on pypi! :)

[jlaehne]

I would like to reorganize the metadata for SEM and TEM related signals as proposed in: https://github.com/hyperspy/hyperspy/issues/2095

The major version and splitting would be the right moment to address this issue!

(I have added it to the checklist above)

[ericpre]

Because of https://github.com/hyperspy/hyperspy/pull/3016, we should release 1.7.2 ASAP. I updated the summary above.

[jlaehne]

Because of hyperspy/hyperspy#3016, we should release 1.7.2 ASAP. I updated the summary above.

See hyperspy/hyperspy#3018

[ericpre]

To track the failure of libraries depending on hyperspy: https://github.com/hyperspy/hyperspy-extensions-list/actions?query=branch%3Aadd_release_next_major

[CSSFrancis]

Do we have an idea of when we want to shoot for a 2.0.0 release? We would like to release a 1.0.0 version of pyxem at the same time or shortly after. I can devote a fair bit of time to this over the next couple of months if necessary to get this moving. I think that at the very latest we should make sure that the 2.0.0 release occurs before M&M in July.

[jlaehne]

I think the main issue is that we still need to split the EELS/EDS and Holography libraries into extension packages to do all the reorganization in a single major release. See discussion above.

Also hyperspy/hyperspy#2095 (metadata conventions) would need to be addressed before a major release, there should not be much to do on the HyperSpy side, but it would entail a bit of work on the rosettasciio end.

In any case, I do support not letting that trail too long, though my capacities in the coming months are a bit limited.

(I edited the todo-list above)

[ericpre]

I would suggest to make a patch release soon, because the current hyperspy released version is broken for lazy generating random number generator with the recent dask release. There should be easy workaround around and it shouldn't concern that much user code but it would be to make a patch release. I have updated the first comment in this issue to track the PRs that need/could be completed.

[jlaehne]

I would suggest to make a patch release soon, because the current hyperspy released version is broken for lazy generating random number generator with the recent dask release. There should be easy workaround around and it shouldn't concern that much user code but it would be to make a patch release. I have updated the first comment in this issue to track the PRs that need/could be completed.

I added a list of PRs merged into RnMajor in the past months that would make sense to add to a patch release already.

[ericpre]

After checking the list above, I suspect that the major and patch branch have diverge significantly, that it may be a pain to correctly backport these changes... and I am not sure that they are worth the effort. From the list above I would backport only hyperspy/hyperspy#3084. @jlaehne, what do you think?

[jlaehne]

Yes, that is probably the easiest and most important one.

We should then probably focus on getting the 2.0 release out soon as well.

[CSSFrancis]

We should then probably focus on getting the 2.0 release out soon as well.

Is there anything that I can do to help this along? I would like to get this figured out sooner rather than later. I can work on cleaning up my code additions but there are some points that I would like some additional input before finishing.

I don't know if I am good enough git wizard to split out the eels and eds functions but I can help organize and set up the extensions after the fact. There are a couple of meetings/conferences/hyperspy workshops coming up and it would be good to get some of these changes in and packaged before then.

[jlaehne]

Is there anything that I can do to help this along? I would like to get this figured out sooner rather than later. I can work on cleaning up my code additions but there are some points that I would like some additional input before finishing.

Well, if you feel comfortable with any of the proposed API changes mentioned above, you could help get that of the table. Also, it would be good to release RosettaSciIO soon: I created a tracking issue https://github.com/hyperspy/rosettasciio/issues/99 and there are still a number of improvements that could be done there.

I don't know if I am good enough git wizard to split out the eels and eds functions but I can help organize and set up the extensions after the fact. There are a couple of meetings/conferences/hyperspy workshops coming up and it would be good to get some of these changes in and packaged before then.

Once RosettaSciIO is released, we need to decide whether we keep the full split for v2.0 or whether we actually wait with the split out of eels, eds and holography for a v3.0. It would be nice to make the full split now, but at the pace that we are at currently that would delay v2.0 quite a bit. The IO split was more work than anticipated, but then we also got a lot of things much cleaner on both the HyperSpy and RosettaSciIO side along the way.

[CSSFrancis]

At this point I think we should start thinking about releasing a 2.0.0 without splitting out the EDS, EELS and Holography. The last major release was over a year ago at this point and there are a fair number of improvements since then.

I've edit the post above to include what I think v2.0.0 should have included.

Looking at these I realize that I might be part of the problem (rather than part of the solution) I will try to make sure all of my open PR's are passing and maybe we can figure out how to best review them as some of them are a bit more involved. Honestly some of them could very easily be added in a minor release as well.

I propose that 3.0.0 includes the EDS, EELS and Holography split and a more formalized metadata structure. We should start looking for someone with interest to take over as a lead developer for that package as well. More consistent and smaller releases would probably do some good looking forward. And all of the splitting should help with that quite a bit.

[magnunor]

I made a pull request to address the - review rechunk argument of lazy functions.: https://github.com/hyperspy/hyperspy/pull/3166

[CSSFrancis]

I might try to start splitting the EELS/EDS as well as the holography unless someone else wants to give it a shot or someone says that we should wait. It shouldn't be terribly difficult to accomplish (hopefully) and then that will bring us a fair bit closer to a 2.0.0 release.

[CSSFrancis]

I might try to start splitting the EELS/EDS as well as the holography unless someone else wants to give it a shot or someone says that we should wait. It shouldn't be terribly difficult to accomplish (hopefully) and then that will bring us a fair bit closer to a 2.0.0 release.

Most of the errors in splitting this result from the pretty heavy use of the EDS_SEM signal https://github.com/hyperspy/hyperspy/blob/70ca0762a332a7239a4c1eb75001062ab6c7495d/hyperspy/datasets/example_signals.py#L2-L3 for testing the model functions. The create_model function for the EDSSignal is a bit difficult to replicate. An easy solution would be to depend on the EELS/EDS package for testing and then we could maybe slowly replace some of those tests.

[jlaehne]

I might try to start splitting the EELS/EDS as well as the holography unless someone else wants to give it a shot or someone says that we should wait. It shouldn't be terribly difficult to accomplish (hopefully) and then that will bring us a fair bit closer to a 2.0.0 release.

Francisco also wanted to work on the EELS/EDS splitting when he finds the time - so maybe you could first start out with the holography part?

[CSSFrancis]

@jlaehne that's not a bad idea. Splitting the Holography should be rather easy.

My original goal of releasing 2.0.0 before M&M next week isn't looking terribly promising but we could maybe see about a release candidate which includes things like #3148.

[CSSFrancis]

So this actually seems like most things are pretty much done...

The big things left to do are:

• [ ] Split the EELS and EDS
• [ ] Split the Holography
• [ ] Clean up the map function
• [ ] Merge a bunch of outstanding code :)

Beyond that we probably should try to ship something as there are lots of big things that are just waiting on this release.

@francisco-dlp or @ericpre were you thinking about tackling the splitting of the EELS and EDS code?

I was going to clean up the map function today which (hopefully) shouldn't be too terribly difficult.

Otherwise I will make another attempt at splitting things once I finish up with map function.