Optview rewrite

[lamkina]

Purpose

This PR introduces a new version of OptView that uses an MVC design pattern and PtQt6, built on the pyOptSparse history file api. This new version will fully replace the old version of OptView and add PyQt6 and matplotlib as dependencies if installed with the pip install .[optview] tag. OpenMDAO case recorder files will no longer be supported in the new version. Features include:

• Updated user interface.
• Command line interface for adding files on startup.
• Support for multiple history files.
• Support for plotting major and minor iterations.
• Ability to add multiple plots and configure them individually.
• Optimization problem metadata viewer with searchable options list.
• Support for multiple plotting tabs that can support up to three subplots each.
• Default niceplots formatting for matplotlib.
• Support for saving plots as pdf, pgf, jpeg, png, and eps using the matplotlib Qt5Agg backend.
• Plot configuration toolbar from the matplotlib Qt5Agg backend.
• All classes and functions are fully documented with docstrings and comments!

Expected time until merged

This PR is quite large and introduces a lot of postprocessing functionality that did not previously exist. This PR will take a long time and likely sit as a draft for a while as people test it out and give feedback.

Type of change

• [ ] Bugfix (non-breaking change which fixes an issue)
• [x] New feature (non-breaking change which adds functionality)
• [ ] Breaking change (non-backwards-compatible fix or feature)
• [ ] Code style update (formatting, renaming)
• [ ] Refactoring (no functional changes, no API changes)
• [ ] Documentation update
• [ ] Maintenance update
• [ ] Other (please describe)

This change will not affect the main functionality of pyOptSparse, but will completely replace the old version of OptView.

Testing

This PR introduces a working version of the new OptView, but still needs lots of test coverage. I plan to add tests for each function in the future, but for now, to check out the functionality, do these steps:

1. Navigate to the postprocessing directory.
2. Run python OptView.py in a terminal.

Checklist

• [x] I have run flake8 and black to make sure the code adheres to PEP-8 and is consistently formatted
• [ ] I have run unit and regression tests which pass locally with my changes
• [ ] I have added new tests that prove my fix is effective or that my feature works
• [ ] I have added necessary documentation

[codecov[bot]]

Codecov Report

Merging #291 (9500b1e) into main (0577f0b) will decrease coverage by 11.76%. The diff coverage is n/a.

@@             Coverage Diff             @@
##             main     #291       +/-   ##
===========================================
- Coverage   84.17%   72.41%   -11.77%     
===========================================
  Files          22       22               
  Lines        3331     3331               
===========================================
- Hits         2804     2412      -392     
- Misses        527      919      +392     

Impacted Files	Coverage Δ
pyoptsparse/pySNOPT/pySNOPT.py	12.92% <0.00%> (-76.93%)	:arrow_down:
pyoptsparse/pyNLPQLP/pyNLPQLP.py	25.68% <0.00%> (-66.98%)	:arrow_down:
pyoptsparse/pyOpt_solution.py	57.77% <0.00%> (-42.23%)	:arrow_down:
pyoptsparse/pyOpt_utils.py	60.46% <0.00%> (-7.91%)	:arrow_down:
pyoptsparse/pyOpt_history.py	77.22% <0.00%> (-5.80%)	:arrow_down:
pyoptsparse/pyIPOPT/pyIPOPT.py	92.50% <0.00%> (-3.34%)	:arrow_down:
pyoptsparse/pyOpt_optimizer.py	83.40% <0.00%> (-1.82%)	:arrow_down:
pyoptsparse/pyOpt_optimization.py	78.19% <0.00%> (-0.81%)	:arrow_down:

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

[ewu63]

Some initial issues when testing:

• optview is not installed as an CLI program, so users have to manually type python <path-to-optview>
• Instead of passing --files <files> the users should just be able to pass a list of files without typing --files. Since that's the only argument, it can be made into a positional argument rather than keyword argument.
• Finally, I get

  FileNotFoundError: [Errno 2] No such file or directory: 'assets/pyOptSparse_logo.png'
  

  You probably need to 1) install the logo along with the rest of the package, and 2) set the path as relative paths.

[lamkina]

I made optview a cli tool and it should work by just typing optview in the command line. There is no flag for adding history files on the command line anymore. You can now just use optview <files>.

I had to make a few edits to setup.py to find the post processing packages and add the cli tool.

Also, I changed all the file paths to be relative and they were working locally, so I think that is also fixed.

[marcomangano]

Nice stuff! I have tested it really quickly and I am experiencing some issues already:

• (minor thing, but it would be good to have a quick way for the user to check if a file is already loaded in any new window - I got a bit confused when I first opened the gui)
• On ubuntu 20.04, when the gui crashes, some "ghosts" icons remain on the sidebar
• The major/minor iteration check is not working for my hst files, and the plots are showing the full linesearch history (I receive a message that says that the var/function has not major or minor attributes)
• Some crashes I experienced:
  • I was playing with data to be added to the same plot, and it randomly failed with the following message: File "/home/mmangano/.pyenv/versions/adflow-test/lib/python3.8/site-packages/matplotlib/axes/_base.py", line 519, in _plot_args raise ValueError(f"x has {ncx} columns but y has {ncy} columns") ValueError: x has 7 columns but y has 117 columns. I was definitely carelessly clicking around, but in the worst case scenario the plot should reset or the request should be ignored, instead of crashing. I will try to reproduce this to give you an MWE
  • I can't plot the constraints of my optimization, because optview crashes with the following message: Traceback (most recent call last): File "/home/mmangano/repos_mdolab/pyoptsparse/pyoptsparse/postprocessing/sub_MVCs/variables/y_controller.py", line 163, in add_var_to_plot self._plot_model.plot() File "/home/mmangano/repos_mdolab/pyoptsparse/pyoptsparse/postprocessing/sub_MVCs/plotting/plot_model.py", line 104, in plot y_var.file.get_var_data(y_var) File "/home/mmangano/repos_mdolab/pyoptsparse/pyoptsparse/postprocessing/utils/data_structures.py", line 104, in get_var_data scale = np.array(var_info["scale"]) if scale_opt else np.ones(len(var_info["scale"])) TypeError: object of type 'int' has no len()
• Selecting single DVs within a set, as done in the previous versions, is very important and should be easy for the user. Maybe having a sub-menu in the add/configure tab would make things easier.
• The legend is not very useful, we should find a way to at least add the indices to the var names. Additionally, some of the matplotlib options for the legend (e.g. location, fontsize, and so on) should be added to the figure options panel. The data label option also seems to apply to the first (or last?) item of the list (see plot below, just one of the vars was renamed to banana)

The issues above lead to the current situation for one of my aerostructural optimization

I can send you an example hst file, some of the stuff above might come from a different optimization formulation.

Anyway, aside this dev issues to be fixed, I think this is really the way to go, thanks a lot for the effort! I like the matplotlib-based customization, the fact that you can create multiple tabs, the tecplot-like selection of the data to be plotted, and other.

I feel that this also opens to more future developments in terms of customization and data viz. One thing that I think would be very useful is to allow the user to set some "macros", so that e.g the same set of plots can be replicated quickly for different files or on different sessions.

[lamkina]

Thanks for the feedback and detailed bug reports. Some notes from your feedback:

• I'm planning to add a way to select individual variables when a DV/function is vectorized.
• I've also noticed the ghost icons and I'm looking into the reasons (Only happens on linux not Mac OS)
• I can add a file list display on the main window to show the loaded files quickly.
• Great comment about the legend config options, I will add a feature that allows for greater control over the legend formatting.
• The major/minor iterations and scale option errors were caused by inconsistencies with the history api that need to be caught in OptView. (I am aware of them).
• The macro concept is a great, but it will have to be a long-term development.

[lamkina]

Just pushed new commits with the following updates and features:

• Data is now loaded when the file is loaded. This takes slightly more time up front but makes most other operations quicker.
• Variable groups are now parsed by index and displayed with a color code in the plot configuration window.
• Custom labels can be added for variables that will also show up in the legend.
• When removing variables from the plot, the background color will reset to the system default.
• You can now add and remove batches of variables at once using multi-selection.
• When searching variables or metadata options, the data will now auto-filter to show you only relevant results.
• The variables are now sorted first by name and second by index
• The variable index is now displayed along with the variable.
• A small display was added to the main window to see what files are already loaded.
• You will now be prompted to enter a refresh interval time when the auto-refresh feature is turned on.
• Many bug fixes to minor/major iteration handling and plotting.

[lamkina]

Lates commit update:

• Added legend configuration tab to the matplotlib toolbar figure options.
• Legend can be made draggable in the configuration options.
• Refactored the File and Variable data structures to be more useful and efficient.
• Added dark mode using the qdarkstyle package. I couldn't find a better way to do this because Qt doesn't inherit the system theme for MacOS. If we don't want the extra dependency this can easily be removed and stick with light mode only.

[lamkina]

New Additions:

• Keyboard shortcuts
  • ctrl+f: Figure parameters dialog
  • ctrl+s: Save figure dialog
  • ctrl+o: Load file dialog
  • ctrl+p: Add plot
  • ctrl+t: Format figure with a "tight layout"
  • ctrl+shift+h: Re-center the figure
• Plot order is now configurable
  • The order of subplots can be changed by dragging and dropping the list items.
  • This can also be done by selecting the plot list item you want to move and pressing ctrl+up and ctrl+down